fix: merge dev into main

PLAN.md

@@ -0,0 +1,189 @@
+# Silksoup MTL Automated Post-Editing (APE) Engine & Enhancement Plan
+
+## 1. Overview & Objectives
+
+Enhance Silksoup with an Automated Post-Editing (APE) pipeline designed to repair Machine Translation (MTL) quirks, typos, dropped pronouns, and inconsistent named entities across multi-chapter web novels.
+
+### Key Goals
+1. **Context-Aware Consistency**: Maintain character names, genders, speech cadences, and world terms across hundreds of chapters.
+2. **Title Deduplication**: Ensure chapter titles appear exactly once as the canonical `<h1>` and are never duplicated into chapter body paragraphs.
+3. **Strict Translator Note Isolation**: Detect translator commentary, cultural notes, and glossary inserts and wrap them strictly in semantic `<blockquote><p><strong>TL Note:</strong> ...</p></blockquote>`.
+4. **Token & Cost Efficiency**: Exploit DeepSeek's automatic prefix KV-caching (up to 90% input token discount) and entity-filtered chunking.
+5. **Dual-File Library Model**: Keep regular and fixed versions distinct (`<slug>.epub` vs. `<slug>-fixed.epub`), displayed in the UI as stacked 0-margin paired entries with dark azure/blue styling for fixed editions.
+
+---
+
+## 2. Context Management Architecture
+
+### A. Global Lore Sheet (`lore_json`)
+- **Storage**: Persistent in DB (`novel_lore` table) and cached in-memory per novel.
+- **Initialization**: Automatically extracted via LLM from the novel's synopsis and first 2–3 chapters on the first run.
+- **Structure**:
+  ```json
+  {
+    "characters": [
+      {
+        "raw_name": "アルス",
+        "mtl_variants": ["Arus", "Ars", "Arce"],
+        "standard_name": "Ars",
+        "gender": "male",
+        "role": "Protagonist",
+        "speech_style": "Casual, direct, slightly sarcastic"
+      }
+    ],
+    "terms": {
+      "魔力回路": "Mana Circuit",
+      "黒の騎士団": "Black Knights"
+    },
+    "rules": {
+      "keep_honorifics": false,
+      "pov": "Third person limited"
+    }
+  }
+  ```
+- **Chunk-Level Filtering**: Before sending each chunk to the LLM, only entities/terms present in the raw chunk text are injected into the active prompt.
+
+### B. Rolling Scene Context (`rolling_summary_json`)
+- **Storage**: JSON updated after each chapter completion.
+- **Structure**:
+  ```json
+  {
+    "recent_chapters": [
+      {
+        "chapter_index": 14,
+        "title": "Chapter 14: The Descent",
+        "summary": "Ars and Elena entered dungeon level 3. Elena suffered a minor leg injury; they found a temporary safe alcove."
+      },
+      {
+        "chapter_index": 15,
+        "title": "Chapter 15: Safe Haven",
+        "summary": "Ars applies basic first aid to Elena. They discuss their next route towards the teleport gate."
+      }
+    ]
+  }
+  ```
+- **Token Bound**: Kept strictly under 150 tokens per summary to minimize overhead.
+
+---
+
+## 3. Chapter Processing & Text Chunker
+
+### A. Paragraph-Boundary Chunker
+- Splits sanitized chapter text at clean paragraph boundaries (`\n\n` or `</p>`).
+- Target chunk size: **700–1,000 words** (~1,000–1,300 tokens).
+- Preserves all embedded `<img src="...">` and `<hr />` markers.
+
+### B. Single Title Guarantee
+- During chunk 1 pre-processing, matching chapter title strings, leading `<h1>`/`<h2>`, or Markdown `# Chapter Title` lines are stripped from the body content.
+- `epub_creator.py` injects the single authoritative `<h1>{chapter_name}</h1>` at the top of the XHTML chapter.
+
+### C. Translator Notes Formatting
+- Translator notes are identified during APE and output strictly as:
+  ```html
+  <blockquote class="tl-note">
+    <p><strong>TL Note:</strong> Explanation or cultural context.</p>
+  </blockquote>
+  ```
+- Styled in `style/nav.css` with a subtle border and indent.
+
+---
+
+## 4. DeepSeek Prompt & Prefix-Caching Optimization
+
+### A. System Prompt Design
+To maximize DeepSeek prefix caching, the static system prompt remains byte-for-byte identical across all calls:
+
+```
+You are an expert Japanese-to-English literary translator and copyeditor specializing in light novels and web novels. Your goal is to transform raw Machine Translation (MTL) into fluent, natural, and immersive English narrative prose while maintaining 100% factual accuracy to the source material.
+
+CORE OPERATIONAL RULES:
+1. STRICT CONTEXT ALIGNMENT (Zero Hallucination):
+   - Do NOT invent, add, or alter plot points, actions, dialogue beats, or world-building elements.
+   - Do NOT insert extra prose, backstory, or commentary that was not present in the raw MTL text.
+2. FIDELITY-PRESERVING STYLISTIC POLISHING:
+   - Eliminate awkward literal idioms, stiff sentence structures, repetitive word choices, and robotic cadence.
+   - Fix dropped or inverted pronouns based on active characters in the scene context.
+   - Smooth dialogue to match character voice defined in the Active Glossary.
+3. GLOSSARY & ENTITY ENFORCEMENT:
+   - Always strictly honor the provided Active Glossary. Do not translate proper names literally if a standardized equivalent is specified.
+4. TRANSLATOR NOTES:
+   - Format any translator/editor notes strictly as: <blockquote class="tl-note"><p><strong>TL Note:</strong> ...</p></blockquote>
+5. OUTPUT SPECIFICATION:
+   - Output ONLY the polished chapter prose wrapped in standard <p>...</p> tags.
+   - Preserve <img> and <hr /> tags.
+   - Do NOT duplicate the chapter title at the beginning.
+   - Do NOT include markdown code block wrappers (```html) or introductory conversational filler.
+```
+
+### B. Temperature & Tokens
+- `temperature`: `0.15` (prevents hallucination and ensures entity fidelity).
+- `max_tokens`: `4096` per chunk.
+
+---
+
+## 5. Storage & Database Schema
+
+### Updated Tables (`src/database.py`)
+```sql
+-- Chapter page with fixed text cache
+CREATE TABLE IF NOT EXISTS chapter_page (
+    url TEXT PRIMARY KEY,
+    status_code INTEGER,
+    html_text TEXT,
+    clean_html_text TEXT,
+    fixed_html_text TEXT,
+    fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+-- Global Lore and Context storage
+CREATE TABLE IF NOT EXISTS novel_lore (
+    novel_url TEXT PRIMARY KEY,
+    lore_json TEXT,
+    rolling_summary_json TEXT,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+```
+
+---
+
+## 6. File Naming & Library Pairing System
+
+### A. EPUB Output File Naming
+- Regular Scrape: `output/<novel-slug>.epub`
+- Fixed Scrape: `output/<novel-slug>-fixed.epub`
+
+### B. Duplicate Guard on Scrape URL
+- When a user submits a novel URL in the "Scrape Novel" form:
+  - If `<novel-slug>-fixed.epub` already exists in `output/`, the job submission is blocked.
+  - Return notice: `"Novel already exists in library (Fixed version present)."`
+
+### C. Library Stacking Card UI
+- In the Library section:
+  - Regular entries and Fixed entries of the same novel are paired into **sticking entries with 0 margin**:
+    - **Top Entry (Regular)**: Standard dark surface background (`bg-theme-surface`), with `[Download]`, `[Fix]`, and `[Delete]` buttons.
+    - **Bottom Entry (Fixed)**: **Dark azure/blue background** (`#0D1C42`, border `#22396F`, text `#FCF1D0`, badge `#38bdf8`), with `[Download]` and `[Delete]` buttons only (no `[Fix]` button).
+  - If a novel was scraped with the `Fix` radio button enabled from the beginning, it appears directly as the dark azure/blue fixed entry.
+
+---
+
+## 7. Implementation Steps
+
+1. **Database & Schema Updates (`src/database.py`)**:
+   - Add `fixed_html_text` column and `novel_lore` table.
+2. **Prompts & Extraction (`src/prompt.py`)**:
+   - Add `LORE_EXTRACTION_PROMPT`, `APE_SYSTEM_PROMPT`, `CHAPTER_SUMMARY_PROMPT`.
+3. **Core APE Module (`src/ape.py`)**:
+   - Implement `extract_initial_lore()`, `chunk_chapter_text()`, `run_ape_on_chunk()`, `update_rolling_summary()`, `fix_chapter()`.
+4. **Chapter & EPUB Pipeline Integration (`src/chapter.py`, `src/epub_creator.py`)**:
+   - Support `mode="fix"` across chapter scraping and EPUB compilation.
+   - Add `blockquote.tl-note` to `_STYLE`.
+5. **Backend Job & API Routing (`web/job_manager.py`, `web/app.py`)**:
+   - Add `/api/jobs` support for `fix` mode.
+   - Add `/api/epubs/{name}/fix` endpoint to trigger fixing directly from library.
+   - Add duplicate check for existing fixed EPUBs.
+   - Group paired EPUB files in `/api/epubs`.
+6. **Frontend UI Updates (`web/templates/index.html`)**:
+   - Add `Fix` radio option in Scrape form.
+   - Implement stacked 0-margin paired entries in the Library card with dark azure/blue styling for fixed editions.
+7. **Verification & Testing**:
+   - Verify with FastAPI TestClient and mock novel runs.

src/ape.py

@@ -0,0 +1,236 @@
+import json
+import re
+from bs4 import BeautifulSoup
+from .database import db_execute, db_fetch_one
+from .llm import generate_json, generate_text
+from .prompt import (
+    APE_SYSTEM_PROMPT,
+    CHAPTER_SUMMARY_SYSTEM_PROMPT,
+    LORE_EXTRACTION_SYSTEM_PROMPT,
+)
+
+
+def get_or_init_novel_lore(novel_url: str, metadata: dict, initial_sample_html: str = "") -> dict:
+    row = db_fetch_one("SELECT lore_json FROM novel_lore WHERE novel_url = ?", (novel_url,))
+    if row is not None and row[0]:
+        try:
+            return json.loads(row[0])
+        except Exception:
+            pass
+
+    series_meta = metadata.get("metadata", {})
+    description = series_meta.get("description") or ""
+    title = series_meta.get("title") or "Novel"
+
+    soup = BeautifulSoup(initial_sample_html, "lxml")
+    sample_text = soup.get_text(separator=" ", strip=True)[:4000]
+
+    input_text = f"Title: {title}\nDescription: {description}\n\nSample Chapter Content:\n{sample_text}"
+    try:
+        raw_json = generate_json(LORE_EXTRACTION_SYSTEM_PROMPT, input_text)
+        lore = json.loads(raw_json)
+    except Exception:
+        lore = {
+            "characters": [],
+            "terms": {},
+            "rules": {"keep_honorifics": False, "pov": "Third person limited"},
+        }
+
+    lore_str = json.dumps(lore, ensure_ascii=False)
+    db_execute(
+        "INSERT INTO novel_lore (novel_url, lore_json, rolling_summary_json) VALUES (?, ?, ?) ON CONFLICT(novel_url) DO UPDATE SET lore_json=excluded.lore_json, updated_at=CURRENT_TIMESTAMP",
+        (novel_url, lore_str, json.dumps([])),
+    )
+    return lore
+
+
+def get_rolling_summary(novel_url: str) -> list:
+    row = db_fetch_one("SELECT rolling_summary_json FROM novel_lore WHERE novel_url = ?", (novel_url,))
+    if row is not None and row[0]:
+        try:
+            return json.loads(row[0])
+        except Exception:
+            return []
+    return []
+
+
+def update_rolling_summary(novel_url: str, chapter_name: str, chapter_html: str) -> None:
+    soup = BeautifulSoup(chapter_html, "lxml")
+    plain_text = soup.get_text(separator=" ", strip=True)
+    if not plain_text:
+        return
+
+    sample = plain_text[:3500]
+    try:
+        summary_text = generate_text(CHAPTER_SUMMARY_SYSTEM_PROMPT, sample).strip()
+    except Exception:
+        summary_text = f"Events occurred in {chapter_name}."
+
+    existing = get_rolling_summary(novel_url)
+    existing.append({"title": chapter_name, "summary": summary_text})
+    recent = existing[-2:]
+
+    db_execute(
+        "INSERT INTO novel_lore (novel_url, rolling_summary_json) VALUES (?, ?) ON CONFLICT(novel_url) DO UPDATE SET rolling_summary_json=excluded.rolling_summary_json, updated_at=CURRENT_TIMESTAMP",
+        (novel_url, json.dumps(recent, ensure_ascii=False)),
+    )
+
+
+def strip_duplicate_chapter_title(html: str, chapter_title: str) -> str:
+    if not html or not html.strip():
+        return ""
+    soup = BeautifulSoup(html, "lxml")
+    body = soup.body if soup.body else soup
+
+    # Clean title for comparison
+    clean_target = re.sub(r"[^\w\s]", "", chapter_title.lower()).strip()
+    target_words = set(clean_target.split())
+
+    # Check top headings or paragraphs
+    for tag in list(body.find_all(["h1", "h2", "h3", "p", "div"], recursive=False))[:3]:
+        tag_text = tag.get_text(strip=True)
+        clean_tag = re.sub(r"[^\w\s]", "", tag_text.lower()).strip()
+        tag_words = set(clean_tag.split())
+
+        # Exact match or high overlap with chapter title / "chapter X"
+        if clean_target and (clean_tag == clean_target or (len(target_words) >= 2 and target_words.issubset(tag_words))):
+            tag.decompose()
+            continue
+        if re.match(r"^(chapter|episode|prologue|epilogue)\s*\d*", clean_tag, re.I) and len(clean_tag.split()) <= 4:
+            tag.decompose()
+            continue
+
+    return body.decode_contents().strip()
+
+
+def chunk_chapter_text(html: str, target_words: int = 800) -> list[str]:
+    if not html or not html.strip():
+        return []
+
+    soup = BeautifulSoup(html, "lxml")
+    body = soup.body if soup.body else soup
+
+    elements = list(body.find_all(["p", "hr", "img", "blockquote", "aside"], recursive=False))
+    if not elements:
+        # Fallback: split by paragraph or line breaks
+        paragraphs = [p.strip() for p in html.split("\n") if p.strip()]
+        chunks = []
+        curr = []
+        curr_words = 0
+        for p in paragraphs:
+            w_count = len(p.split())
+            if curr_words + w_count > target_words and curr:
+                chunks.append("\n".join(curr))
+                curr = [p]
+                curr_words = w_count
+            else:
+                curr.append(p)
+                curr_words += w_count
+        if curr:
+            chunks.append("\n".join(curr))
+        return chunks
+
+    chunks = []
+    curr_elements = []
+    curr_word_count = 0
+
+    for el in elements:
+        text = el.get_text(strip=True)
+        w_count = len(text.split())
+        if curr_word_count + w_count > target_words and curr_elements:
+            chunk_html = "".join(str(e) for e in curr_elements)
+            chunks.append(chunk_html)
+            curr_elements = [el]
+            curr_word_count = w_count
+        else:
+            curr_elements.append(el)
+            curr_word_count += w_count
+
+    if curr_elements:
+        chunk_html = "".join(str(e) for e in curr_elements)
+        chunks.append(chunk_html)
+
+    return chunks
+
+
+def filter_glossary_for_chunk(global_lore: dict, chunk_text: str) -> dict:
+    if not global_lore:
+        return {}
+
+    lowered_chunk = chunk_text.lower()
+    filtered_chars = []
+    for char in global_lore.get("characters", []):
+        names = [char.get("standard_name", ""), char.get("raw_name", "")] + char.get("mtl_variants", [])
+        if any(n and n.lower() in lowered_chunk for n in names):
+            filtered_chars.append(char)
+
+    filtered_terms = {}
+    for term, trans in global_lore.get("terms", {}).items():
+        if (term and term.lower() in lowered_chunk) or (trans and trans.lower() in lowered_chunk):
+            filtered_terms[term] = trans
+
+    # If no matches, retain top character or default to keep context active
+    if not filtered_chars and global_lore.get("characters"):
+        filtered_chars = global_lore["characters"][:2]
+
+    return {
+        "characters": filtered_chars,
+        "terms": filtered_terms,
+        "rules": global_lore.get("rules", {}),
+    }
+
+
+def clean_llm_html_output(raw_output: str) -> str:
+    cleaned = re.sub(r"^```(?:html)?\s*", "", raw_output.strip(), flags=re.I)
+    cleaned = re.sub(r"\s*```$", "", cleaned)
+    # Ensure any <aside class="tl-note"> is converted to <blockquote>
+    cleaned = re.sub(r"<aside\s+class=[\"']tl-note[\"']>(.*?)</aside>", r"<blockquote class=\"tl-note\"><p><strong>TL Note:</strong> \1</p></blockquote>", cleaned, flags=re.DOTALL | re.I)
+    return cleaned.strip()
+
+
+def fix_chapter(
+    raw_cleaned_html: str,
+    chapter_name: str,
+    novel_url: str,
+    global_lore: dict,
+) -> str:
+    # 1. Deduplicate title from start of body
+    html_without_title = strip_duplicate_chapter_title(raw_cleaned_html, chapter_name)
+
+    # 2. Chunk text
+    chunks = chunk_chapter_text(html_without_title, target_words=800)
+    if not chunks:
+        chunks = [html_without_title] if html_without_title else ["<p></p>"]
+
+    # 3. Format rolling scene context
+    rolling = get_rolling_summary(novel_url)
+    if rolling:
+        context_lines = [f"- {r.get('title', 'Previous')}: {r.get('summary', '')}" for r in rolling]
+        scene_context_str = "Recent Scene Context:\n" + "\n".join(context_lines)
+    else:
+        scene_context_str = f"Scene Context: Opening / early chapter ({chapter_name})."
+
+    fixed_chunks = []
+    for chunk in chunks:
+        active_glossary = filter_glossary_for_chunk(global_lore, chunk)
+        user_prompt = (
+            f"### Active Glossary & Lore\n```json\n{json.dumps(active_glossary, ensure_ascii=False, indent=2)}\n```\n\n"
+            f"### Scene Context\n{scene_context_str}\n\n"
+            f"### Raw MTL Text to Process\n{chunk}"
+        )
+
+        try:
+            polished = generate_text(APE_SYSTEM_PROMPT, user_prompt)
+            fixed_chunk = clean_llm_html_output(polished)
+        except Exception:
+            # Fallback to original chunk if LLM fails on a segment
+            fixed_chunk = clean_llm_html_output(chunk)
+
+        fixed_chunks.append(fixed_chunk)
+
+    final_fixed_html = "\n".join(fixed_chunks)
+
+    # 4. Update rolling summary for next chapters
+    update_rolling_summary(novel_url, chapter_name, final_fixed_html)
+
+    return final_fixed_html

src/chapter.py

@@ -4,6 +4,7 @@ from .parse import robust_fetch
 from .clean_html import clean_novel_html
 from .prompt import HTML_CHAPTER_SANITIZER
 from .llm import generate_text
+from .ape import get_or_init_novel_lore, fix_chapter
 
 
 def iter_chapters(metadata: dict):
@@ -25,7 +26,7 @@ def cache_chapter_hit(url: str, refresh_mode: str = "none") -> str:
     if isinstance(refresh_mode, bool):
         refresh_mode = "all" if refresh_mode else "none"
 
-    if refresh_mode != "all":
+    if refresh_mode not in ("all", "all_refresh"):
         row = db_fetch_one("SELECT html_text FROM chapter_page WHERE url = ?", (url,))
         if row is not None and row[0] is not None:
             tqdm.write(f"Chapter cache hit: {url}")
@@ -65,6 +66,36 @@ def cache_clean_hit(url: str, refresh_mode: str = "none") -> str:
     return sanitized
 
 
+def cache_fixed_hit(
+    url: str,
+    chapter_name: str,
+    novel_url: str,
+    global_lore: dict,
+    refresh_mode: str = "none",
+) -> str:
+    if refresh_mode == "none":
+        row = db_fetch_one("SELECT fixed_html_text FROM chapter_page WHERE url = ?", (url,))
+        if row is not None and row[0] is not None:
+            tqdm.write(f"Fixed HTML cache hit: {url}")
+            return str(row[0])
+
+    # Retrieve clean html first
+    clean_html = cache_clean_hit(url, refresh_mode="none" if refresh_mode != "all" else "all")
+    fixed_html = fix_chapter(
+        raw_cleaned_html=clean_html,
+        chapter_name=chapter_name,
+        novel_url=novel_url,
+        global_lore=global_lore,
+    )
+
+    db_execute(
+        "UPDATE chapter_page SET fixed_html_text = ? WHERE url = ?",
+        (fixed_html, url),
+    )
+
+    return fixed_html
+
+
 def scrape_chapters(
     metadata: dict,
     refresh_mode: str = "none",
@@ -117,3 +148,49 @@ def clean_chapters(
             tqdm.write(f"[SKIP] {chapter['name']}: {e}")
             if progress_callback:
                 progress_callback(stage="cleaning", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")
+
+
+def fix_chapters(
+    metadata: dict,
+    refresh_mode: str = "none",
+    progress_callback=None,
+    cancel_event=None,
+) -> None:
+    chapters = list(iter_chapters(metadata))
+    total = len(chapters)
+    novel_url = metadata.get("novel_url") or ""
+
+    # Sample the first chapter for initial lore extraction if needed
+    first_chapter_sample = ""
+    if chapters:
+        try:
+            first_raw = cache_chapter_hit(chapters[0]["url"], refresh_mode="none")
+            first_chapter_sample = clean_novel_html(first_raw, base_url=chapters[0]["url"])
+        except Exception:
+            pass
+
+    global_lore = get_or_init_novel_lore(novel_url, metadata, first_chapter_sample)
+
+    if progress_callback is None:
+        iterator = tqdm(chapters, desc="Fixing MTL chapters", unit="ch")
+    else:
+        iterator = chapters
+    for idx, chapter in enumerate(iterator, start=1):
+        if cancel_event is not None and cancel_event.is_set():
+            raise InterruptedError("Fixing was cancelled by user")
+        try:
+            cache_fixed_hit(
+                url=chapter["url"],
+                chapter_name=chapter.get("name") or f"Chapter {idx}",
+                novel_url=novel_url,
+                global_lore=global_lore,
+                refresh_mode=refresh_mode,
+            )
+            if progress_callback:
+                progress_callback(stage="fixing", current=idx, total=total, message=chapter.get("name") or chapter.get("url") or "")
+        except InterruptedError:
+            raise
+        except Exception as e:
+            tqdm.write(f"[SKIP] {chapter['name']}: {e}")
+            if progress_callback:
+                progress_callback(stage="fixing", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")

src/database.py

@@ -83,9 +83,23 @@ def init_db() -> None:
                         status_code INTEGER,
                         html_text TEXT,
                         clean_html_text TEXT,
+                        fixed_html_text TEXT,
                         fetched_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
                     )
                 """)
+                cur.execute("""
+                    CREATE TABLE IF NOT EXISTS novel_lore (
+                        novel_url TEXT PRIMARY KEY,
+                        lore_json TEXT,
+                        rolling_summary_json TEXT,
+                        updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
+                    )
+                """)
+                # Handle existing tables missing columns
+                try:
+                    cur.execute("ALTER TABLE chapter_page ADD COLUMN IF NOT EXISTS fixed_html_text TEXT;")
+                except Exception:
+                    pass
             conn.commit()
         finally:
             conn.close()
@@ -108,6 +122,21 @@ def init_db() -> None:
                     status_code INTEGER,
                     html_text TEXT,
                     clean_html_text TEXT,
+                    fixed_html_text TEXT,
                     fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                 )
             """)
+            conn.execute("""
+                CREATE TABLE IF NOT EXISTS novel_lore (
+                    novel_url TEXT PRIMARY KEY,
+                    lore_json TEXT,
+                    rolling_summary_json TEXT,
+                    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+                )
+            """)
+            # Migration check for SQLite
+            cur = conn.cursor()
+            cur.execute("PRAGMA table_info(chapter_page);")
+            columns = [row[1] for row in cur.fetchall()]
+            if "fixed_html_text" not in columns:
+                conn.execute("ALTER TABLE chapter_page ADD COLUMN fixed_html_text TEXT;")

src/epub_creator.py

@@ -42,12 +42,17 @@ hr {
     border-top: 1px solid #ccc;
     margin: 2em 0;
 }
-aside.tl-note {
+aside.tl-note, blockquote.tl-note {
     font-size: 90%;
-    margin: 1em 2em;
-    padding: 0.5em 1em;
-    border-left: 3px solid #ccc;
-    color: #555;
+    margin: 1.5em 1em;
+    padding: 0.6em 1.2em;
+    border-left: 3px solid #888;
+    background-color: #f5f5f5;
+    color: #444;
+}
+blockquote.tl-note p {
+    margin: 0;
+    text-indent: 0;
 }
 """
 
@@ -58,7 +63,16 @@ def _slugify(text: str) -> str:
     return slug or "novel"
 
 
-def _get_clean_html(url: str) -> str:
+def _get_chapter_html(url: str, is_fixed: bool = False) -> str:
+    if is_fixed:
+        row = db_fetch_one("SELECT fixed_html_text, clean_html_text FROM chapter_page WHERE url = ?", (url,))
+        if row is not None:
+            if row[0]:
+                return str(row[0])
+            if row[1]:
+                return str(row[1])
+        raise RuntimeError(f"No fixed or cleaned HTML cached for {url}. Run fix_chapters first.")
+
     row = db_fetch_one("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
     if row is None or row[0] is None:
         raise RuntimeError(
@@ -180,16 +194,22 @@ def create_epub(
     output_dir: str = "output",
     progress_callback=None,
     cancel_event=None,
+    is_fixed: bool = False,
 ) -> str:
     series = metadata.get("metadata", {})
     title = series.get("title") or "Untitled"
+    if is_fixed and not title.endswith(" [Fixed]"):
+        book_title = f"{title} [Fixed]"
+    else:
+        book_title = title
+
     author = series.get("author") or "Unknown"
     language = series.get("language") or "en"
     cover_url = series.get("cover_url")
 
     book = epub.EpubBook()
-    book.set_identifier(_slugify(title))
-    book.set_title(title)
+    book.set_identifier(_slugify(book_title))
+    book.set_title(book_title)
     book.set_language(language)
     book.add_author(author)
 
@@ -242,13 +262,13 @@ def create_epub(
                 continue
 
             try:
-                clean_html = _get_clean_html(url)
+                chapter_html = _get_chapter_html(url, is_fixed=is_fixed)
             except RuntimeError as e:
                 tqdm.write(f"[SKIP] {chapter_name}: {e}")
                 chapter_counter += 1
                 continue
 
-            clean_html = _embed_images(book, clean_html, image_cache, referer=url)
+            chapter_html = _embed_images(book, chapter_html, image_cache, referer=url)
 
             file_name = f"chapter_{chapter_counter}.xhtml"
             epub_chapter = epub.EpubHtml(
@@ -257,7 +277,7 @@ def create_epub(
                 lang=language,
             )
             epub_chapter.add_item(nav_css)
-            epub_chapter.content = f"<h1>{chapter_name}</h1>\n{clean_html}"
+            epub_chapter.content = f"<h1>{chapter_name}</h1>\n{chapter_html}"
 
             book.add_item(epub_chapter)
             sections.append(epub_chapter)
@@ -285,7 +305,9 @@ def create_epub(
     book.spine = spine_structure
 
     os.makedirs(output_dir, exist_ok=True)
-    output_path = os.path.join(output_dir, f"{_slugify(title)}.epub")
+    slug = _slugify(title)
+    filename = f"{slug}-fixed.epub" if is_fixed else f"{slug}.epub"
+    output_path = os.path.join(output_dir, filename)
 
     epub.write_epub(output_path, book, {})
     print(f"Successfully generated: {output_path}")

src/prompt.py

@@ -53,7 +53,7 @@ TAG & FORMATTING RULES:
 
 TRANSLATOR & CONTEXT NOTES:
 1. KEEP informative translator notes that explain cultural context, puns, lore, or honorifics. Format them cleanly as:
-   <aside class="tl-note"><strong>Note:</strong> [Context explanation here]</aside>
+   <blockquote class="tl-note"><p><strong>TL Note:</strong> [Context explanation here]</p></blockquote>
 2. REMOVE translator banter, jokes, meme bloopers, editor rants, or off-topic conversational commentary.
 
 STRICT REMOVAL RULES:
@@ -64,5 +64,84 @@ STRICT REMOVAL RULES:
 STRICT FIDELITY RULES:
 1. DO NOT summarize, paraphrase, rewrite, or correct the author's prose or character dialogue.
 2. DO NOT delete legitimate story paragraphs.
-3. Output ONLY the sanitized HTML body content (<p>, <hr />, <img>, <aside>). Do not include <html>, <head>, <body> wrappers, or markdown code fences (```html).
+3. Output ONLY the sanitized HTML body content (<p>, <hr />, <img>, <blockquote>). Do not include <html>, <head>, <body> wrappers, or markdown code fences (```html).
+"""
+
+
+LORE_EXTRACTION_SYSTEM_PROMPT = """
+You are an expert light novel entity extraction and lore analysis engine.
+
+YOUR TASK:
+Analyze the provided novel description and initial chapters to extract key characters, proper terminology, and narrative rules into a standardized JSON glossary.
+
+EXTRACTION SCHEMA:
+{
+  "characters": [
+    {
+      "raw_name": "Original Japanese or Romanized Name",
+      "mtl_variants": ["Variant1", "Variant2"],
+      "standard_name": "Standardized English Name",
+      "gender": "male | female | unknown",
+      "role": "Protagonist | Ally | Antagonist | Supporting",
+      "speech_style": "Brief description of dialogue tone"
+    }
+  ],
+  "terms": {
+    "Original Term or MTL Term": "Standardized English Term"
+  },
+  "rules": {
+    "keep_honorifics": false,
+    "pov": "First-person | Third-person limited | Third-person omniscient"
+  }
+}
+
+RULES:
+1. Identify true recurring character names vs generic nouns.
+2. Standardize spelling variants (e.g., "Arus", "Arce", "Ars" -> "Ars").
+3. Identify faction names, skill/magic names, and location names.
+4. Output ONLY valid raw JSON.
+"""
+
+
+APE_SYSTEM_PROMPT = """
+You are an expert Japanese-to-English literary translator and copyeditor specializing in light novels and web novels. Your goal is to transform raw Machine Translation (MTL) into fluent, natural, and immersive English narrative prose while maintaining 100% factual accuracy to the source material.
+
+CORE OPERATIONAL RULES:
+1. STRICT CONTEXT ALIGNMENT (Zero Hallucination):
+   - Do NOT invent, add, or alter plot points, actions, dialogue beats, or world-building elements.
+   - Do NOT insert extra prose, backstory, or commentary that was not present or implied in the raw MTL text.
+   - If a sentence is completely incomprehensible, reconstruct the most logical meaning using surrounding paragraph context and the provided Glossary/Lore sheet.
+
+2. FIDELITY-PRESERVING STYLISTIC POLISHING:
+   - Eliminate common MTL artifacts: awkward literal idioms, stiff sentence structures, repetitive word choices, and robotic cadence.
+   - Fix dropped or inverted pronouns (Japanese subject-dropping often causes MTL to confuse "he/she/they/it"). Cross-reference active characters in the scene to ensure correct pronoun usage.
+   - Smooth out dialogue to sound natural when spoken aloud, matching character voice and tone as defined in the Glossary.
+   - Preserve narrative perspective (1st-person POV vs. 3rd-person limited) consistently throughout the scene.
+
+3. GLOSSARY & ENTITY ENFORCEMENT:
+   - Always strictly honor the provided Active Glossary & Character Sheet.
+   - Do not translate proper names, places, or technique names literally if an official/standardized English equivalent is specified in the Glossary.
+
+4. TRANSLATOR NOTES:
+   - Format any informative translator/editor notes strictly as:
+     <blockquote class="tl-note"><p><strong>TL Note:</strong> [Context explanation]</p></blockquote>
+   - Strip out off-topic translator jokes, memes, or donation links.
+
+5. OUTPUT SPECIFICATION:
+   - Output ONLY the polished chapter prose wrapped in standard <p>...</p> tags, with <hr /> for scene breaks and intact <img> tags.
+   - Do NOT duplicate the chapter title at the beginning.
+   - Do NOT include markdown code block wrappers (```html) or introductory conversational text.
+"""
+
+
+CHAPTER_SUMMARY_SYSTEM_PROMPT = """
+You are an expert story summarization engine.
+
+YOUR TASK:
+Read the provided chapter prose and generate an extremely concise 2-sentence summary of the main plot points, current location, active characters, and ending state.
+
+CONSTRAINTS:
+- Maximum 150 words / 200 tokens.
+- Factual and direct. Focus on character actions and scene transitions.
+- Output ONLY the summary text, with no introductory text or labels.
 """

web/app.py

@@ -1,6 +1,7 @@
 import asyncio
 import json
 import os
+import re
 from contextlib import asynccontextmanager
 from pathlib import Path
 
@@ -9,6 +10,8 @@ from fastapi.responses import HTMLResponse, FileResponse, StreamingResponse
 from fastapi.staticfiles import StaticFiles
 from fastapi.templating import Jinja2Templates
 
+from src.database import db_fetch_one, db_fetch_all, init_db
+from src.epub_creator import _slugify
 from .job_manager import (
     jobs,
     job_queues,
@@ -24,6 +27,7 @@ OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
+    init_db()
     yield
     cancel_all_jobs()
 
@@ -42,24 +46,81 @@ async def index(request: Request):
     return templates.TemplateResponse(request, "index.html")
 
 
+def _find_novel_url_for_epub(filename: str) -> str | None:
+    base_slug = _slugify(re.sub(r"(-fixed)?\.epub$", "", filename, flags=re.I))
+    rows = db_fetch_all("SELECT url, json_text FROM main_page WHERE json_text IS NOT NULL")
+    for row in rows:
+        url, json_str = row[0], row[1]
+        try:
+            data = json.loads(json_str)
+            title = data.get("metadata", {}).get("title") or ""
+            if _slugify(title) == base_slug:
+                return url
+        except Exception:
+            continue
+    return None
+
+
 @app.post("/api/jobs")
 async def start_job(payload: dict):
     url = (payload.get("url") or "").strip()
     if not url or not (url.startswith("http://") or url.startswith("https://")):
         raise HTTPException(status_code=400, detail="Valid URL required")
 
+    # Check if a fixed version of this novel already exists in the library
+    row = db_fetch_one("SELECT json_text FROM main_page WHERE url = ?", (url,))
+    if row is not None and row[0]:
+        try:
+            data = json.loads(row[0])
+            title = data.get("metadata", {}).get("title") or ""
+            slug = _slugify(title)
+            fixed_file = OUTPUT_DIR / f"{slug}-fixed.epub"
+            if fixed_file.exists():
+                raise HTTPException(
+                    status_code=409,
+                    detail="Novel already exists in library (Fixed version present).",
+                )
+        except HTTPException:
+            raise
+        except Exception:
+            pass
+
     refresh_mode = payload.get("refresh_mode")
     if refresh_mode is None:
         force_refresh = bool(payload.get("force_refresh", False))
         refresh_mode = "all" if force_refresh else "none"
-    elif refresh_mode not in ("none", "llm", "all"):
+    elif refresh_mode not in ("none", "llm", "all", "fix"):
         refresh_mode = "none"
 
-    job_id = create_job(url, refresh_mode=refresh_mode, output_dir=str(OUTPUT_DIR))
+    fix = bool(payload.get("fix", False)) or (refresh_mode == "fix")
+
+    job_id = create_job(url, refresh_mode=refresh_mode, fix=fix, output_dir=str(OUTPUT_DIR))
     asyncio.create_task(run_job(job_id))
     return {"job_id": job_id, "status": "queued"}
 
 
+@app.post("/api/epubs/{filename}/fix")
+async def fix_library_epub(filename: str):
+    safe = Path(filename).name
+    target = OUTPUT_DIR / safe
+    if not target.exists() or not target.is_file():
+        raise HTTPException(status_code=404, detail="EPUB file not found")
+
+    if "-fixed.epub" in safe.lower():
+        raise HTTPException(status_code=400, detail="This EPUB is already a fixed version.")
+
+    novel_url = _find_novel_url_for_epub(safe)
+    if not novel_url:
+        raise HTTPException(
+            status_code=404,
+            detail="Source novel URL not found in cache. Re-scrape the novel to fix.",
+        )
+
+    job_id = create_job(novel_url, refresh_mode="none", fix=True, output_dir=str(OUTPUT_DIR))
+    asyncio.create_task(run_job(job_id))
+    return {"job_id": job_id, "status": "queued", "novel_url": novel_url}
+
+
 @app.post("/api/jobs/{job_id}/cancel")
 async def stop_job(job_id: str):
     job = jobs.get(job_id)
@@ -154,8 +215,13 @@ async def list_epubs():
     if OUTPUT_DIR.exists():
         for p in sorted(OUTPUT_DIR.glob("*.epub"), key=lambda x: x.stat().st_mtime, reverse=True):
             stat = p.stat()
+            name = p.name
+            is_fixed = "-fixed.epub" in name.lower() or " [fixed]" in name.lower()
+            base_key = re.sub(r"(-fixed)?\.epub$", "", name, flags=re.I).strip("-_")
             files.append({
-                "name": p.name,
+                "name": name,
+                "base_key": base_key,
+                "is_fixed": is_fixed,
                 "size": stat.st_size,
                 "size_human": _human_size(stat.st_size),
                 "mtime": stat.st_mtime,

web/job_manager.py

@@ -8,7 +8,7 @@ from src.database import init_db
 from src.parse import cache_hit
 from src.clean_html import clean_novel_html
 from src.metadata import extract_metadata_json
-from src.chapter import clean_chapters
+from src.chapter import clean_chapters, fix_chapters, scrape_chapters
 from src.epub_creator import create_epub
 
 jobs: dict[str, dict] = {}
@@ -38,12 +38,13 @@ def _push(job_id: str, event: dict):
             pass
 
 
-def create_job(url: str, refresh_mode: str = "none", output_dir: str = "output") -> str:
+def create_job(url: str, refresh_mode: str = "none", fix: bool = False, output_dir: str = "output") -> str:
     job_id = uuid.uuid4().hex[:12]
     jobs[job_id] = {
         "id": job_id,
         "url": url,
         "refresh_mode": refresh_mode,
+        "fix": bool(fix),
         "output_dir": output_dir,
         "status": "queued",
         "stage": "queued",
@@ -93,6 +94,7 @@ async def run_job(job_id: str):
         return
     url = job["url"]
     refresh_mode = job.get("refresh_mode", "none")
+    is_fix_job = bool(job.get("fix") or refresh_mode == "fix")
     output_dir = job["output_dir"]
     cancel_evt = cancel_events.get(job_id)
     job["status"] = "running"
@@ -111,7 +113,7 @@ async def run_job(job_id: str):
         _push(job_id, {"stage": "fetching", "current": 0, "total": 1, "message": "Fetching index page..."})
         job["logs"].append(f"Fetching {url}")
 
-        html = await asyncio.to_thread(cache_hit, url, refresh_mode)
+        html = await asyncio.to_thread(cache_hit, url, "none" if is_fix_job else refresh_mode)
         if cancel_evt and cancel_evt.is_set():
             raise InterruptedError("Cancelled after fetch")
 
@@ -120,7 +122,7 @@ async def run_job(job_id: str):
         _push(job_id, {"stage": "metadata", "current": 0, "total": 1, "message": "Extracting metadata via LLM..."})
         job["logs"].append("Extracting metadata")
 
-        metadata = await asyncio.to_thread(extract_metadata_json, cleaned, url, refresh_mode)
+        metadata = await asyncio.to_thread(extract_metadata_json, cleaned, url, "none" if is_fix_job else refresh_mode)
         metadata["novel_url"] = url
 
         if cancel_evt and cancel_evt.is_set():
@@ -128,6 +130,24 @@ async def run_job(job_id: str):
 
         volumes = metadata.get("volumes", [])
         total_chapters = sum(len(v.get("chapters", [])) for v in volumes)
+
+        if is_fix_job:
+            _push(job_id, {"stage": "scraping", "current": 0, "total": total_chapters, "message": f"Ensuring {total_chapters} chapters are cached"})
+            await asyncio.to_thread(scrape_chapters, metadata, refresh_mode if refresh_mode != "fix" else "none", _cb, cancel_evt)
+
+            if cancel_evt and cancel_evt.is_set():
+                raise InterruptedError("Cancelled before APE fix")
+
+            _push(job_id, {"stage": "fixing", "current": 0, "total": total_chapters, "message": f"Fixing MTL for {total_chapters} chapters"})
+            job["logs"].append(f"Starting APE MTL repair on {total_chapters} chapters")
+            await asyncio.to_thread(fix_chapters, metadata, "none", _cb, cancel_evt)
+
+            if cancel_evt and cancel_evt.is_set():
+                raise InterruptedError("Cancelled before EPUB building")
+
+            _push(job_id, {"stage": "epub", "current": 0, "total": total_chapters, "message": "Building Fixed EPUB..."})
+            output_path = await asyncio.to_thread(create_epub, metadata, output_dir, _cb, cancel_evt, is_fixed=True)
+        else:
             _push(job_id, {"stage": "cleaning", "current": 0, "total": total_chapters, "message": f"Cleaning {total_chapters} chapters"})
             job["logs"].append(f"Found {len(volumes)} volumes, {total_chapters} chapters")
 
@@ -137,7 +157,7 @@ async def run_job(job_id: str):
                 raise InterruptedError("Cancelled before EPUB building")
 
             _push(job_id, {"stage": "epub", "current": 0, "total": total_chapters, "message": "Building EPUB..."})
-        output_path = await asyncio.to_thread(create_epub, metadata, output_dir, _cb, cancel_evt)
+            output_path = await asyncio.to_thread(create_epub, metadata, output_dir, _cb, cancel_evt, is_fixed=False)
 
         job["result_path"] = output_path
         job["status"] = "completed"

web/templates/index.html

@@ -1,5 +1,5 @@
 <!DOCTYPE html>
-<html lang="en">
+<html lang="en" class="dark">
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
@@ -7,17 +7,6 @@
 <link rel="icon" type="image/x-icon" href="/static/favicon.ico">
 <link rel="icon" type="image/png" href="/static/icon.png">
 <link rel="apple-touch-icon" href="/static/icon.png">
-<script>
-(function() {
-  const saved = localStorage.getItem('silksoup_theme');
-  const prefersDark = window.matchMedia('(prefers-color-scheme: dark)').matches;
-  if (saved === 'dark' || (!saved && prefersDark)) {
-    document.documentElement.classList.add('dark');
-  } else {
-    document.documentElement.classList.remove('dark');
-  }
-})();
-</script>
 <script src="https://cdn.tailwindcss.com"></script>
 <script>
 tailwind.config = {
@@ -51,26 +40,6 @@ tailwind.config = {
 </script>
 <style>
 :root {
-  --bg: #fafafa;
-  --surface: #ffffff;
-  --surface-subtle: #f4f4f5;
-  --border: #e4e4e7;
-  --border-strong: #d4d4d8;
-  --primary: #18181b;
-  --primary-hover: #27272a;
-  --primary-fg: #ffffff;
-  --secondary: #71717a;
-  --text: #09090b;
-  --text-muted: #71717a;
-  --danger: #dc2626;
-  --danger-bg: #fef2f2;
-  --danger-border: #fecaca;
-  --log-bg: #09090b;
-  --log-border: #27272a;
-  --log-text: #f4f4f5;
-}
-
-html.dark {
   --bg: #09090b;
   --surface: #121215;
   --surface-subtle: #1c1c21;
@@ -95,7 +64,7 @@ html.dark {
 ::-webkit-scrollbar-track { background: transparent; }
 </style>
 </head>
-<body class="bg-theme-bg text-theme-text min-h-screen flex flex-col font-sans antialiased selection:bg-zinc-800 selection:text-white dark:selection:bg-zinc-200 dark:selection:text-zinc-900">
+<body class="bg-theme-bg text-theme-text min-h-screen flex flex-col font-sans antialiased selection:bg-zinc-800 selection:text-white">
 <header class="border-b border-theme-border bg-theme-surface sticky top-0 z-30">
   <div class="max-w-4xl mx-auto px-4 py-3.5 flex items-center justify-between">
     <div class="flex items-center gap-3">
@@ -103,16 +72,12 @@ html.dark {
       <div>
         <div class="flex items-center gap-2">
           <span class="font-semibold text-base leading-none text-theme-text">Silksoup</span>
-          <span id="headerBadge" class="hidden text-xs font-medium px-2 py-0.5 rounded bg-emerald-50 dark:bg-emerald-950/60 text-emerald-700 dark:text-emerald-400 border border-emerald-200 dark:border-emerald-800/80">Active</span>
+          <span id="headerBadge" class="hidden text-xs font-medium px-2 py-0.5 rounded bg-emerald-950/60 text-emerald-400 border border-emerald-800/80">Active</span>
         </div>
         <p class="text-sm text-theme-muted mt-0.5">Novel to EPUB pipeline</p>
       </div>
     </div>
     <div class="flex items-center gap-2.5">
-      <button id="themeToggleBtn" onclick="toggleTheme()" type="button" aria-label="Toggle theme" class="text-sm text-theme-muted hover:text-theme-text px-3 py-1.5 rounded-md border border-theme-border hover:bg-theme-subtle hover:border-theme-border-strong transition flex items-center gap-1.5">
-        <span id="themeIcon" class="w-4 h-4 flex items-center justify-center"></span>
-        <span id="themeText" class="hidden sm:inline">Theme</span>
-      </button>
       <button id="notifyBtn" onclick="toggleNotifications()" type="button" aria-label="Desktop notifications" class="text-sm text-theme-muted hover:text-theme-text px-3 py-1.5 rounded-md border border-theme-border hover:bg-theme-subtle hover:border-theme-border-strong transition flex items-center gap-1.5">
         <svg class="w-4 h-4" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
           <path d="M6 8a6 6 0 0 1 12 0c0 7 3 9 3 9H3s3-2 3-9"/>
@@ -157,25 +122,37 @@ html.dark {
             <input type="radio" name="refreshMode" value="none" checked class="mt-0.5 text-theme-primary border-theme-border focus:ring-0">
             <div class="flex flex-col">
               <span class="text-sm font-medium text-theme-text">Full Cache</span>
-              <span class="text-xs text-theme-muted mt-0.5">Reuse HTML and parsed results</span>
+              <span class="text-xs text-theme-muted mt-0.5">Reuse HTML & AI results</span>
             </div>
           </label>
           <label class="flex items-start gap-3 p-3.5 rounded-md border border-theme-border bg-theme-subtle/50 hover:border-theme-border-strong cursor-pointer select-none transition">
             <input type="radio" name="refreshMode" value="llm" class="mt-0.5 text-theme-primary border-theme-border focus:ring-0">
             <div class="flex flex-col">
               <span class="text-sm font-medium text-theme-text">Re-run AI</span>
-              <span class="text-xs text-theme-muted mt-0.5">Reuse HTML, re-sanitize text</span>
+              <span class="text-xs text-theme-muted mt-0.5">Reuse HTML, re-sanitize</span>
             </div>
           </label>
           <label class="flex items-start gap-3 p-3.5 rounded-md border border-theme-border bg-theme-subtle/50 hover:border-theme-border-strong cursor-pointer select-none transition">
             <input type="radio" name="refreshMode" value="all" class="mt-0.5 text-theme-primary border-theme-border focus:ring-0">
             <div class="flex flex-col">
               <span class="text-sm font-medium text-theme-text">Full Refresh</span>
-              <span class="text-xs text-theme-muted mt-0.5">Bypass cache and fetch from web</span>
+              <span class="text-xs text-theme-muted mt-0.5">Bypass cache & fetch fresh</span>
             </div>
           </label>
         </div>
       </div>
+      <div>
+        <label class="flex items-start gap-3 p-3.5 rounded-md border border-[#22396F] bg-[#0D1C42]/60 hover:bg-[#0D1C42] cursor-pointer select-none transition">
+          <input id="fixCheck" type="checkbox" class="mt-0.5 rounded border-[#22396F] bg-[#090e1a] text-sky-400 focus:ring-0">
+          <div class="flex flex-col">
+            <div class="flex items-center gap-2">
+              <span class="text-sm font-medium text-[#FCF1D0]">Fix (Automated Post-Editing)</span>
+              <span class="text-xs font-mono uppercase px-1.5 py-0.2 rounded bg-sky-950 text-sky-300 border border-sky-700/80">APE</span>
+            </div>
+            <span class="text-xs text-sky-300/70 mt-0.5">Repair bad MTL, fix pronouns, enforce consistent character names, and format notes</span>
+          </div>
+        </label>
+      </div>
       <div class="flex justify-end pt-1">
         <button id="startBtn" type="button" class="w-full sm:w-auto rounded-md bg-theme-primary hover:bg-theme-primary-hover text-theme-primary-fg text-sm font-medium px-5 py-2.5 transition flex items-center justify-center gap-2 shadow-sm">
           <svg id="btnSpinner" class="hidden w-4 h-4 animate-spin text-current" viewBox="0 0 24 24" fill="none">
@@ -255,7 +232,7 @@ html.dark {
       </div>
     </div>
 
-    <div id="epubList" class="space-y-2.5"></div>
+    <div id="epubList" class="space-y-3"></div>
     <div id="epubEmpty" class="text-sm text-theme-muted py-8 text-center hidden">
       No EPUBs found. Scrape a novel to populate your library.
     </div>
@@ -273,8 +250,8 @@ html.dark {
         LLM extracts chronological volumes and chapter URLs.
       </div>
       <div class="p-3 rounded-md bg-theme-subtle/70 border border-theme-border/70">
-        <span class="text-sm font-medium text-theme-text block mb-1">3. Sanitize Content</span>
-        Cleans site bloat and structures semantic chapter content.
+        <span class="text-sm font-medium text-theme-text block mb-1">3. Sanitize & APE</span>
+        Fixes bad MTL, typos, entities, and formats notes into blockquotes.
       </div>
       <div class="p-3 rounded-md bg-theme-subtle/70 border border-theme-border/70">
         <span class="text-sm font-medium text-theme-text block mb-1">4. Build EPUB</span>
@@ -297,24 +274,9 @@ let evtSource = null;
 let allEpubs = [];
 let originalTitle = document.title;
 
-const sunSvg = `<svg class="w-3.5 h-3.5" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.93 4.93l1.41 1.41M17.66 17.66l1.41 1.41M2 12h2M20 12h2M6.34 17.66l-1.41 1.41M19.07 4.93l-1.41 1.41"/></svg>`;
-const moonSvg = `<svg class="w-3.5 h-3.5" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 3a6 6 0 0 0 9 9 9 9 0 1 1-9-9Z"/></svg>`;
-
-function updateThemeUI() {
-  const isDark = document.documentElement.classList.contains('dark');
-  $('#themeIcon').innerHTML = isDark ? sunSvg : moonSvg;
-  $('#themeText').textContent = isDark ? 'Light' : 'Dark';
-}
-
-function toggleTheme() {
-  const isDark = document.documentElement.classList.toggle('dark');
-  localStorage.setItem('silksoup_theme', isDark ? 'dark' : 'light');
-  updateThemeUI();
-}
-
 function updateNotifyButton() {
   if (Notification.permission === 'granted') {
-    $('#notifyBtn').classList.add('text-emerald-700', 'dark:text-emerald-400', 'border-emerald-300', 'dark:border-emerald-800');
+    $('#notifyBtn').classList.add('text-emerald-400', 'border-emerald-800');
     $('#notifyText').textContent = 'Notifications on';
   }
 }
@@ -343,7 +305,7 @@ function showToast(message, type = 'info') {
   const container = $('#toastContainer');
   const toast = document.createElement('div');
   const borderClass = type === 'error' ? 'border-theme-danger-border bg-theme-danger-bg text-theme-danger' :
-                      type === 'success' ? 'border-emerald-300 dark:border-emerald-800 bg-emerald-50 dark:bg-emerald-950 text-emerald-900 dark:text-emerald-200' :
+                      type === 'success' ? 'border-emerald-800 bg-emerald-950 text-emerald-200' :
                       'border-theme-border bg-theme-surface text-theme-text';
   toast.className = `pointer-events-auto rounded-md border px-3.5 py-2.5 text-sm shadow-md flex items-center gap-2.5 transition-opacity duration-150 opacity-0 ${borderClass}`;
   toast.textContent = message;
@@ -394,10 +356,11 @@ $('#startBtn').addEventListener('click', async () => {
   try {
     const selectedMode = document.querySelector('input[name="refreshMode"]:checked');
     const refresh_mode = selectedMode ? selectedMode.value : 'none';
+    const fix = $('#fixCheck') ? $('#fixCheck').checked : false;
     const res = await fetch('/api/jobs', {
       method: 'POST',
       headers: {'Content-Type': 'application/json'},
-      body: JSON.stringify({ url, refresh_mode })
+      body: JSON.stringify({ url, refresh_mode, fix })
     });
     if (!res.ok) {
       const j = await res.json();
@@ -426,7 +389,7 @@ function showActive() {
   $('#jobStage').textContent = 'queued';
   $('#jobStatusBadge').textContent = 'running';
   $('#jobResult').classList.add('hidden');
-  $('#statusPulse').className = 'w-2 h-2 rounded-full bg-emerald-500';
+  $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-500';
   $('#activeJob').scrollIntoView({ behavior: 'smooth', block: 'nearest' });
 }
 
@@ -476,27 +439,27 @@ function connectSSE(jobId) {
     $('#jobCounter').textContent = `${evt.current} / ${evt.total}`;
     $('#jobProgress').style.width = pct + '%';
 
-    if (evt.stage === 'scraping' || evt.stage === 'cleaning' || evt.stage === 'epub') {
+    if (evt.stage === 'scraping' || evt.stage === 'cleaning' || evt.stage === 'fixing' || evt.stage === 'epub') {
       document.title = `(${pct}%) Silksoup`;
     }
 
     if (evt.stage === 'failed') {
       $('#jobStatusBadge').textContent = 'failed';
-      $('#statusPulse').className = 'w-2 h-2 rounded-full bg-rose-500';
+      $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
       $('#cancelJobBtn').classList.add('hidden');
       document.title = `Failed — Silksoup`;
       showToast(`Job failed: ${evt.message}`, 'error');
     }
     if (evt.stage === 'cancelled') {
       $('#jobStatusBadge').textContent = 'cancelled';
-      $('#statusPulse').className = 'w-2 h-2 rounded-full bg-amber-500';
+      $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
       $('#cancelJobBtn').classList.add('hidden');
       document.title = `Silksoup`;
-      showToast('Scraping stopped. Completed chapters cached.', 'info');
+      showToast('Process stopped. Cached chapters remain safe.', 'info');
     }
     if (evt.stage === 'completed') {
       $('#jobStatusBadge').textContent = 'completed';
-      $('#statusPulse').className = 'w-2 h-2 rounded-full bg-emerald-500';
+      $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-500';
       $('#cancelJobBtn').classList.add('hidden');
       document.title = `Completed — Silksoup`;
       notify('Silksoup EPUB Ready', evt.message || 'Book compiled successfully.');
@@ -522,7 +485,7 @@ async function refreshJob() {
   const j = await res.json();
   $('#jobStatusBadge').textContent = j.status;
   if (j.status === 'completed') {
-    $('#statusPulse').className = 'w-2 h-2 rounded-full bg-emerald-500';
+    $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-500';
     $('#cancelJobBtn').classList.add('hidden');
     if (j.result_path) {
       const name = j.result_path.split(/[\\/]/).pop();
@@ -532,11 +495,11 @@ async function refreshJob() {
     }
   }
   if (j.status === 'cancelled') {
-    $('#statusPulse').className = 'w-2 h-2 rounded-full bg-amber-500';
+    $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
     $('#cancelJobBtn').classList.add('hidden');
   }
   if (j.status === 'failed') {
-    $('#statusPulse').className = 'w-2 h-2 rounded-full bg-rose-500';
+    $('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
     $('#cancelJobBtn').classList.add('hidden');
     $('#jobLog').textContent += `\n[ERROR] ${j.error || j.message}\n`;
   }
@@ -585,6 +548,25 @@ function filterEpubs() {
   renderEpubs(filtered);
 }
 
+async function triggerFix(encodedName) {
+  const name = decodeURIComponent(encodedName);
+  try {
+    const res = await fetch(`/api/epubs/${encodeURIComponent(name)}/fix`, { method: 'POST' });
+    if (!res.ok) {
+      const j = await res.json();
+      throw new Error(j.detail || 'Failed to start fix job');
+    }
+    const data = await res.json();
+    currentJobId = data.job_id;
+    localStorage.setItem('silksoup_active_job', currentJobId);
+    showActive();
+    connectSSE(currentJobId);
+    showToast('Fix process started for ' + name, 'success');
+  } catch(e) {
+    showToast(e.message, 'error');
+  }
+}
+
 function renderEpubs(list) {
   const container = $('#epubList');
   container.innerHTML = '';
@@ -594,11 +576,31 @@ function renderEpubs(list) {
   }
   $('#epubEmpty').classList.add('hidden');
 
+  // Group EPUBs by base novel slug
+  const groups = {};
   for (const f of list) {
-    const row = document.createElement('div');
-    row.className = 'flex items-center justify-between gap-3.5 rounded-md border border-theme-border bg-theme-subtle/50 px-4 py-3 hover:border-theme-border-strong transition';
+    const key = f.base_key || f.name.replace(/(-fixed)?\.epub$/i, '').trim().toLowerCase();
+    if (!groups[key]) groups[key] = { regular: null, fixed: null };
+    if (f.is_fixed) {
+      groups[key].fixed = f;
+    } else {
+      groups[key].regular = f;
+    }
+  }
 
+  for (const key of Object.keys(groups)) {
+    const g = groups[key];
+    const groupWrapper = document.createElement('div');
+    groupWrapper.className = 'flex flex-col';
+
+    const hasBoth = g.regular && g.fixed;
+
+    if (g.regular) {
+      const f = g.regular;
       const cleanTitle = f.name.replace(/\.epub$/i, '').replace(/[-_]/g, ' ');
+      const row = document.createElement('div');
+      const roundedClass = hasBoth ? 'rounded-t-md border-b-0' : 'rounded-md';
+      row.className = `flex items-center justify-between gap-3.5 border border-theme-border bg-theme-subtle/50 px-4 py-3 hover:border-theme-border-strong transition ${roundedClass}`;
 
       row.innerHTML = `
         <div class="flex items-center gap-3 min-w-0 flex-1">
@@ -613,13 +615,49 @@ function renderEpubs(list) {
           </div>
         </div>
         <div class="flex items-center gap-2 shrink-0">
-        <a href="/api/epubs/${encodeURIComponent(f.name)}" download class="text-sm px-3.5 py-1.5 rounded-md bg-theme-primary hover:bg-theme-primary-hover text-theme-primary-fg font-medium transition shadow-sm">Download</a>
+          <a href="/api/epubs/${encodeURIComponent(f.name)}" download class="text-sm px-3 py-1.5 rounded-md bg-theme-primary hover:bg-theme-primary-hover text-theme-primary-fg font-medium transition shadow-sm">Download</a>
+          <button type="button" onclick="triggerFix('${encodeURIComponent(f.name)}')" class="text-sm px-3 py-1.5 rounded-md border border-sky-800 bg-sky-950/40 hover:bg-sky-900/60 text-sky-300 font-medium transition">Fix</button>
           <div class="relative inline-block" id="delWrap_${CSS.escape(f.name)}">
-          <button type="button" onclick="confirmDelete('${encodeURIComponent(f.name)}')" class="text-sm px-3 py-1.5 rounded-md border border-theme-border hover:border-theme-danger-border text-theme-muted hover:text-theme-danger transition">Delete</button>
+            <button type="button" onclick="confirmDelete('${encodeURIComponent(f.name)}')" class="text-sm px-2.5 py-1.5 rounded-md border border-theme-border hover:border-theme-danger-border text-theme-muted hover:text-theme-danger transition">Delete</button>
           </div>
         </div>
       `;
-    container.appendChild(row);
+      groupWrapper.appendChild(row);
+    }
+
+    if (g.fixed) {
+      const f = g.fixed;
+      const cleanTitle = f.name.replace(/\.epub$/i, '').replace(/[-_]/g, ' ');
+      const row = document.createElement('div');
+      const roundedClass = hasBoth ? 'rounded-b-md' : 'rounded-md';
+      row.className = `flex items-center justify-between gap-3.5 border border-[#22396F] bg-[#0D1C42] px-4 py-3 text-[#FCF1D0] transition shadow-sm ${roundedClass}`;
+
+      row.innerHTML = `
+        <div class="flex items-center gap-3 min-w-0 flex-1">
+          <svg class="w-4 h-4 text-sky-400 shrink-0" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
+            <path d="M4 19.5v-15A2.5 2.5 0 0 1 6.5 2H20v20H6.5a2.5 2.5 0 0 1-2.5-2.5Z"/>
+            <path d="M6 6h10"/>
+            <path d="M6 10h10"/>
+          </svg>
+          <div class="min-w-0 flex-1">
+            <div class="flex items-center gap-2">
+              <span class="truncate text-sm font-medium text-[#FCF1D0] capitalize" title="${cleanTitle}">${cleanTitle}</span>
+              <span class="text-xs font-mono uppercase px-1.5 py-0.2 rounded bg-sky-950 text-sky-300 border border-sky-700/80">Fixed</span>
+            </div>
+            <div class="text-xs text-sky-300/70 font-mono mt-0.5">${f.size_human} • ${new Date(f.mtime * 1000).toLocaleDateString()} ${new Date(f.mtime * 1000).toLocaleTimeString([], {hour: '2-digit', minute: '2-digit'})}</div>
+          </div>
+        </div>
+        <div class="flex items-center gap-2 shrink-0">
+          <a href="/api/epubs/${encodeURIComponent(f.name)}" download class="text-sm px-3.5 py-1.5 rounded-md bg-[#FCF1D0] hover:bg-white text-[#0D1C42] font-semibold transition shadow-sm">Download</a>
+          <div class="relative inline-block" id="delWrap_${CSS.escape(f.name)}">
+            <button type="button" onclick="confirmDelete('${encodeURIComponent(f.name)}')" class="text-sm px-2.5 py-1.5 rounded-md border border-[#22396F] hover:border-rose-900/80 text-sky-300/70 hover:text-rose-400 transition">Delete</button>
+          </div>
+        </div>
+      `;
+      groupWrapper.appendChild(row);
+    }
+
+    container.appendChild(groupWrapper);
   }
 }
 
@@ -674,7 +712,6 @@ async function checkStoredJob() {
   }
 }
 
-updateThemeUI();
 updateNotifyButton();
 checkStoredJob();
 loadEpubs();