Compare commits
10
Commits
5678bd87b8
...
bd3d978216
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bd3d978216 | ||
|
|
2581e01e3b | ||
|
|
648c15a7ef | ||
|
|
361ff407f0 | ||
|
|
948c826acf | ||
|
|
0314ebb931 | ||
|
|
2ebd54cc6f | ||
|
|
cf9a6cfd0d | ||
|
|
9f15215c26 | ||
|
|
00d751f99f |
@@ -0,0 +1,5 @@
|
|||||||
|
LLM_API_KEY=your_api_key_here
|
||||||
|
LLM_BASE_URL=https://api.openai.com/v1
|
||||||
|
LLM_MODEL=ds/deepseek-v4-flash
|
||||||
|
DATABASE_URL=postgresql://user:password@localhost:5432/silksoup
|
||||||
|
OUTPUT_DIR=output
|
||||||
@@ -4,3 +4,4 @@ __pycache__/
|
|||||||
*.pyc
|
*.pyc
|
||||||
sqlite.db
|
sqlite.db
|
||||||
.env
|
.env
|
||||||
|
output/
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
build-essential curl libxml2-dev libxslt1-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY requirements.txt .
|
||||||
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
RUN mkdir -p output
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
CMD ["uvicorn", "web.app:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
# Silksoup
|
||||||
|
|
||||||
|
> Turn online web novels and light novels into clean, beautifully formatted EPUB e-books with a single click or command.
|
||||||
|
|
||||||
|
Silksoup scrapes web novel index pages, uses an OpenAI-compatible LLM to detect volumes and chapters, cleans out site ads and junk, downloads illustrations, and packages everything into an offline `.epub` file ready for your e-reader, phone, or tablet.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- **Dual Modes:** Use the visual **Web Dashboard** or the terminal **CLI tool**—both share the same engine.
|
||||||
|
- **Smart Chapter Extraction:** Uses AI to understand complex table of contents, volumes, and reading order.
|
||||||
|
- **Clean Reading Experience:** Strips navigation links, donation banners, and website watermarks while preserving important translator notes, formatting, and character dialogue.
|
||||||
|
- **Image & Illustration Support:** Downloads cover art and embedded chapter illustrations (supports WEBP, JPEG, PNG, GIF) with anti-hotlinking headers.
|
||||||
|
- **Smart Caching (PostgreSQL & SQLite):** Stores downloaded chapters so re-runs and updates don't re-download what you already have.
|
||||||
|
- **Self-Hostable & Docker Ready:** Run it locally or host it on a VPS behind WireGuard / VPN.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start (Web App)
|
||||||
|
|
||||||
|
### Option A: Using Docker (Recommended for Servers / VPS)
|
||||||
|
|
||||||
|
1. **Clone the repository:**
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/your-username/silksoup.git
|
||||||
|
cd silksoup
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Set up your environment variables:**
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
```
|
||||||
|
Open `.env` and configure your OpenAI-compatible LLM:
|
||||||
|
```env
|
||||||
|
LLM_API_KEY=your_api_key_here
|
||||||
|
LLM_BASE_URL=https://api.openai.com/v1
|
||||||
|
LLM_MODEL=ds/deepseek-v4-flash
|
||||||
|
|
||||||
|
# Optional: Connect to your PostgreSQL database (leave blank to use local SQLite)
|
||||||
|
DATABASE_URL=postgresql://user:password@localhost:5432/silksoup
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Start the service:**
|
||||||
|
```bash
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Open your browser:**
|
||||||
|
Go to `http://localhost:8000` (or `http://YOUR_SERVER_IP:8000`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Option B: Running Locally with Python
|
||||||
|
|
||||||
|
1. **Install dependencies:**
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
# or with uv:
|
||||||
|
uv pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Configure `.env`** (see step 2 above).
|
||||||
|
|
||||||
|
3. **Start the web server:**
|
||||||
|
```bash
|
||||||
|
python server.py
|
||||||
|
```
|
||||||
|
Open `http://localhost:8000` in your browser.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Web Dashboard Guide
|
||||||
|
|
||||||
|
1. **Enter Novel URL:** Paste the main index or table-of-contents link of the novel into the input box.
|
||||||
|
2. **Start Scraping:** Click **"Start Scraping -> EPUB"**.
|
||||||
|
3. **Live Progress:** Watch real-time progress as chapters are fetched, cleaned, and compiled into the EPUB.
|
||||||
|
4. **Download & Manage:** Once complete, click **"Download EPUB"** or manage your compiled books directly from the built-in **Library**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI Usage
|
||||||
|
|
||||||
|
If you prefer working from the terminal, run `main.py` directly:
|
||||||
|
|
||||||
|
### CLI Syntax:
|
||||||
|
```bash
|
||||||
|
python main.py <url> [-o OUTPUT] [--refresh]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flags & Options:
|
||||||
|
|
||||||
|
| Option / Flag | Type | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `url` | *Positional (Required)* | The main index / table-of-contents URL of the novel. |
|
||||||
|
| `-o`, `--output` | *Optional* | Output directory for the compiled EPUB file (default: `./output`). |
|
||||||
|
| `--refresh` | *Optional (Flag)* | Force refresh cached pages and metadata instead of using SQLite/PostgreSQL cache. |
|
||||||
|
| `-h`, `--help` | *Optional (Flag)* | Show command-line help message and exit. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Where Are My EPUB Files?
|
||||||
|
|
||||||
|
All compiled `.epub` files are saved directly inside the `output/` directory in the project folder:
|
||||||
|
- **Host Path:** `./output/<novel-title>.epub`
|
||||||
|
- **Docker Path:** `/app/output/<novel-title>.epub`
|
||||||
|
|
||||||
|
You can move these files to your e-reader (Kindle, Kobo, Boox) or integrate the folder with media servers like **Calibre-Web**, **Kavita**, or **Audiobookshelf**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration Reference
|
||||||
|
|
||||||
|
All settings can be configured inside your `.env` file:
|
||||||
|
|
||||||
|
| Variable | Description | Default |
|
||||||
|
|---|---|---|
|
||||||
|
| `LLM_API_KEY` | **Required.** Your API key for the OpenAI-compatible LLM. | *None* |
|
||||||
|
| `LLM_BASE_URL` | **Required.** Base URL of any OpenAI-compatible API endpoint. | *None* |
|
||||||
|
| `LLM_MODEL` | **Required.** Model identifier used for extraction & cleaning. | *None* |
|
||||||
|
| `DATABASE_URL` | PostgreSQL connection string. If omitted, uses local `sqlite.db`. | *None (SQLite)* |
|
||||||
|
| `OUTPUT_DIR` | Directory where compiled EPUBs are stored. | `output` |
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
services:
|
||||||
|
app:
|
||||||
|
build: .
|
||||||
|
container_name: silksoup
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:8000:8000"
|
||||||
|
env_file:
|
||||||
|
- .env
|
||||||
|
environment:
|
||||||
|
- DATABASE_URL=${DATABASE_URL:-}
|
||||||
|
- LLM_API_KEY=${LLM_API_KEY}
|
||||||
|
- LLM_BASE_URL=${LLM_BASE_URL}
|
||||||
|
- LLM_MODEL=${LLM_MODEL}
|
||||||
|
- OUTPUT_DIR=/app/output
|
||||||
|
volumes:
|
||||||
|
- ./output:/app/output
|
||||||
|
restart: unless-stopped
|
||||||
@@ -1,18 +1,49 @@
|
|||||||
|
import argparse
|
||||||
|
from pathlib import Path
|
||||||
from src.database import init_db
|
from src.database import init_db
|
||||||
from src.metadata import extract_metadata_json
|
from src.metadata import extract_metadata_json
|
||||||
from src.parse import cache_hit
|
from src.parse import cache_hit
|
||||||
from src.clean_html import clean_novel_html
|
from src.clean_html import clean_novel_html
|
||||||
from src.chapter import clean_chapters
|
from src.chapter import clean_chapters
|
||||||
|
from src.epub_creator import create_epub
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Scrape light novel and compile EPUB."
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"url",
|
||||||
|
type=str,
|
||||||
|
help="URL of the novel table of contents / index page",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-o",
|
||||||
|
"--output",
|
||||||
|
type=str,
|
||||||
|
default=str(Path(__file__).resolve().parent / "output"),
|
||||||
|
help="Output directory for the compiled EPUB (default: ./output)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--refresh",
|
||||||
|
action="store_true",
|
||||||
|
help="Force refresh cached pages/metadata instead of using SQLite cache",
|
||||||
|
)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
|
args = parse_args()
|
||||||
init_db()
|
init_db()
|
||||||
|
|
||||||
page_url = "https://hikkicave.site/story/i-became-friends-with-the-second-cutest-girl-in-my-class-wn/"
|
html_content = clean_novel_html(
|
||||||
|
cache_hit(args.url, force_refresh=args.refresh)
|
||||||
html_content = clean_novel_html(cache_hit(page_url))
|
)
|
||||||
metadata = extract_metadata_json(html_content, page_url)
|
metadata = extract_metadata_json(
|
||||||
clean_chapters(metadata)
|
html_content, args.url, force_refresh=args.refresh
|
||||||
|
)
|
||||||
|
clean_chapters(metadata, force_refresh=args.refresh)
|
||||||
|
create_epub(metadata, output_dir=args.output)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
beautifulsoup4==4.15.0
|
||||||
|
lxml==6.1.2
|
||||||
|
curl-cffi==0.16.1
|
||||||
|
ebooklib==0.20
|
||||||
|
Pillow==12.3.0
|
||||||
|
tqdm==4.70.0
|
||||||
|
openai==3.3.1
|
||||||
|
python-dotenv==1.2.3
|
||||||
|
fastapi==0.115.0
|
||||||
|
uvicorn[standard]==0.30.6
|
||||||
|
jinja2==3.1.4
|
||||||
|
psycopg[binary]==3.2.3
|
||||||
|
python-multipart==0.0.9
|
||||||
|
httpx==0.28.1
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
import uvicorn
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
uvicorn.run("web.app:app", host="0.0.0.0", port=8000, reload=False)
|
||||||
+62
-52
@@ -1,15 +1,9 @@
|
|||||||
import sqlite3
|
from tqdm import tqdm
|
||||||
from google import genai
|
from .database import db_execute, db_fetch_one
|
||||||
from google.genai import types
|
|
||||||
from dotenv import load_dotenv
|
|
||||||
from .database import DB_PATH
|
|
||||||
from .parse import robust_fetch
|
from .parse import robust_fetch
|
||||||
from .clean_html import clean_novel_html
|
from .clean_html import clean_novel_html
|
||||||
from .prompt import HTML_CHAPTER_SANITIZER
|
from .prompt import HTML_CHAPTER_SANITIZER
|
||||||
|
from .llm import generate_text
|
||||||
|
|
||||||
load_dotenv()
|
|
||||||
client = genai.Client()
|
|
||||||
|
|
||||||
|
|
||||||
def iter_chapters(metadata: dict):
|
def iter_chapters(metadata: dict):
|
||||||
@@ -24,80 +18,96 @@ def iter_chapters(metadata: dict):
|
|||||||
|
|
||||||
|
|
||||||
def sanitize_chapter_html(html: str) -> str:
|
def sanitize_chapter_html(html: str) -> str:
|
||||||
response = client.models.generate_content(
|
return generate_text(HTML_CHAPTER_SANITIZER, html)
|
||||||
model="gemini-3.6-flash",
|
|
||||||
contents=html,
|
|
||||||
config=types.GenerateContentConfig(
|
|
||||||
system_instruction=HTML_CHAPTER_SANITIZER,
|
|
||||||
temperature=0.0,
|
|
||||||
response_mime_type="text/plain",
|
|
||||||
tools=None,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return response.text.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def cache_chapter_hit(url: str) -> str:
|
def cache_chapter_hit(url: str, force_refresh: bool = False) -> str:
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
if not force_refresh:
|
||||||
cursor = conn.cursor()
|
row = db_fetch_one("SELECT html_text FROM chapter_page WHERE url = ?", (url,))
|
||||||
cursor.execute("SELECT html_text FROM chapter_page WHERE url = ?", (url,))
|
|
||||||
row = cursor.fetchone()
|
|
||||||
if row is not None and row[0] is not None:
|
if row is not None and row[0] is not None:
|
||||||
print(f"Chapter cache hit: {url}")
|
tqdm.write(f"Chapter cache hit: {url}")
|
||||||
return str(row[0])
|
return str(row[0])
|
||||||
|
|
||||||
status_code, html = robust_fetch(url)
|
status_code, html = robust_fetch(url)
|
||||||
if status_code != 200:
|
if status_code != 200:
|
||||||
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
|
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
|
||||||
|
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
db_execute(
|
||||||
conn.execute(
|
"INSERT INTO chapter_page (url, status_code, html_text) VALUES (?, ?, ?) ON CONFLICT(url) DO UPDATE SET status_code=excluded.status_code, html_text=excluded.html_text, fetched_at=CURRENT_TIMESTAMP",
|
||||||
"INSERT OR REPLACE INTO chapter_page (url, status_code, html_text) VALUES (?, ?, ?)",
|
|
||||||
(url, status_code, html),
|
(url, status_code, html),
|
||||||
)
|
)
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
return html
|
return html
|
||||||
|
|
||||||
|
|
||||||
def cache_clean_hit(url: str) -> str:
|
def cache_clean_hit(url: str, force_refresh: bool = False) -> str:
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
if not force_refresh:
|
||||||
cursor = conn.cursor()
|
row = db_fetch_one("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
|
||||||
cursor.execute("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
|
|
||||||
row = cursor.fetchone()
|
|
||||||
if row is not None and row[0] is not None:
|
if row is not None and row[0] is not None:
|
||||||
print(f"Clean HTML cache hit: {url}")
|
tqdm.write(f"Clean HTML cache hit: {url}")
|
||||||
return str(row[0])
|
return str(row[0])
|
||||||
|
|
||||||
raw_html = cache_chapter_hit(url)
|
raw_html = cache_chapter_hit(url, force_refresh=force_refresh)
|
||||||
pre_cleaned = clean_novel_html(raw_html, base_url=url)
|
pre_cleaned = clean_novel_html(raw_html, base_url=url)
|
||||||
sanitized = sanitize_chapter_html(pre_cleaned)
|
sanitized = sanitize_chapter_html(pre_cleaned)
|
||||||
|
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
db_execute(
|
||||||
conn.execute(
|
|
||||||
"UPDATE chapter_page SET clean_html_text = ? WHERE url = ?",
|
"UPDATE chapter_page SET clean_html_text = ? WHERE url = ?",
|
||||||
(sanitized, url),
|
(sanitized, url),
|
||||||
)
|
)
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
return sanitized
|
return sanitized
|
||||||
|
|
||||||
|
|
||||||
def scrape_chapters(metadata: dict) -> None:
|
def scrape_chapters(
|
||||||
for chapter in iter_chapters(metadata):
|
metadata: dict,
|
||||||
print(f"[{chapter['volume']}] {chapter['name']} -> {chapter['url']}")
|
force_refresh: bool = False,
|
||||||
|
progress_callback=None,
|
||||||
|
cancel_event=None,
|
||||||
|
) -> None:
|
||||||
|
chapters = list(iter_chapters(metadata))
|
||||||
|
total = len(chapters)
|
||||||
|
if progress_callback is None:
|
||||||
|
iterator = tqdm(chapters, desc="Scraping chapters", unit="ch")
|
||||||
|
else:
|
||||||
|
iterator = chapters
|
||||||
|
for idx, chapter in enumerate(iterator, start=1):
|
||||||
|
if cancel_event is not None and cancel_event.is_set():
|
||||||
|
raise InterruptedError("Scraping was cancelled by user")
|
||||||
try:
|
try:
|
||||||
cache_chapter_hit(chapter["url"])
|
cache_chapter_hit(chapter["url"], force_refresh=force_refresh)
|
||||||
|
if progress_callback:
|
||||||
|
progress_callback(stage="scraping", current=idx, total=total, message=chapter.get("name") or chapter.get("url") or "")
|
||||||
|
except InterruptedError:
|
||||||
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[SKIP] {chapter['name']}: {e}")
|
tqdm.write(f"[SKIP] {chapter['name']}: {e}")
|
||||||
continue
|
if progress_callback:
|
||||||
|
progress_callback(stage="scraping", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")
|
||||||
|
|
||||||
|
|
||||||
def clean_chapters(metadata: dict) -> None:
|
def clean_chapters(
|
||||||
for chapter in iter_chapters(metadata):
|
metadata: dict,
|
||||||
print(f"[{chapter['volume']}] Cleaning: {chapter['name']}")
|
force_refresh: bool = False,
|
||||||
|
progress_callback=None,
|
||||||
|
cancel_event=None,
|
||||||
|
) -> None:
|
||||||
|
chapters = list(iter_chapters(metadata))
|
||||||
|
total = len(chapters)
|
||||||
|
if progress_callback is None:
|
||||||
|
iterator = tqdm(chapters, desc="Cleaning chapters", unit="ch")
|
||||||
|
else:
|
||||||
|
iterator = chapters
|
||||||
|
for idx, chapter in enumerate(iterator, start=1):
|
||||||
|
if cancel_event is not None and cancel_event.is_set():
|
||||||
|
raise InterruptedError("Cleaning was cancelled by user")
|
||||||
try:
|
try:
|
||||||
cache_clean_hit(chapter["url"])
|
cache_clean_hit(chapter["url"], force_refresh=force_refresh)
|
||||||
|
if progress_callback:
|
||||||
|
progress_callback(stage="cleaning", current=idx, total=total, message=chapter.get("name") or chapter.get("url") or "")
|
||||||
|
except InterruptedError:
|
||||||
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[SKIP] {chapter['name']}: {e}")
|
tqdm.write(f"[SKIP] {chapter['name']}: {e}")
|
||||||
continue
|
if progress_callback:
|
||||||
|
progress_callback(stage="cleaning", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")
|
||||||
|
|||||||
+85
-2
@@ -1,14 +1,98 @@
|
|||||||
|
import os
|
||||||
import sqlite3
|
import sqlite3
|
||||||
|
from contextlib import contextmanager
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
SCRIPT_DIR = Path(__file__).resolve().parent.parent
|
SCRIPT_DIR = Path(__file__).resolve().parent.parent
|
||||||
DB_PATH = str(SCRIPT_DIR / "sqlite.db")
|
DB_PATH = str(SCRIPT_DIR / "sqlite.db")
|
||||||
|
DATABASE_URL = os.environ.get("DATABASE_URL", "").strip()
|
||||||
|
USE_PG = DATABASE_URL.startswith("postgres://") or DATABASE_URL.startswith("postgresql://")
|
||||||
|
|
||||||
|
if USE_PG and DATABASE_URL.startswith("postgres://"):
|
||||||
|
DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _adapt_query(query: str) -> str:
|
||||||
|
if USE_PG:
|
||||||
|
return query.replace("?", "%s")
|
||||||
|
return query
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def get_conn():
|
||||||
|
if USE_PG:
|
||||||
|
import psycopg
|
||||||
|
|
||||||
|
conn = psycopg.connect(DATABASE_URL)
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
else:
|
||||||
|
conn = sqlite3.connect(DB_PATH)
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def db_fetch_one(query: str, params: tuple = ()):
|
||||||
|
q = _adapt_query(query)
|
||||||
|
with get_conn() as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute(q, params)
|
||||||
|
return cur.fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def db_fetch_all(query: str, params: tuple = ()):
|
||||||
|
q = _adapt_query(query)
|
||||||
|
with get_conn() as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute(q, params)
|
||||||
|
return cur.fetchall()
|
||||||
|
|
||||||
|
|
||||||
|
def db_execute(query: str, params: tuple = ()):
|
||||||
|
q = _adapt_query(query)
|
||||||
|
with get_conn() as conn:
|
||||||
|
cur = conn.cursor()
|
||||||
|
cur.execute(q, params)
|
||||||
|
|
||||||
|
|
||||||
def init_db() -> None:
|
def init_db() -> None:
|
||||||
|
if USE_PG:
|
||||||
|
import psycopg
|
||||||
|
|
||||||
|
conn = psycopg.connect(DATABASE_URL)
|
||||||
|
try:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS main_page (
|
||||||
|
url TEXT PRIMARY KEY,
|
||||||
|
status_code INTEGER,
|
||||||
|
html_text TEXT,
|
||||||
|
json_text TEXT,
|
||||||
|
fetched_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
cur.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS chapter_page (
|
||||||
|
url TEXT PRIMARY KEY,
|
||||||
|
status_code INTEGER,
|
||||||
|
html_text TEXT,
|
||||||
|
clean_html_text TEXT,
|
||||||
|
fetched_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
else:
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
with sqlite3.connect(DB_PATH) as conn:
|
||||||
conn.execute("PRAGMA journal_mode=WAL;")
|
conn.execute("PRAGMA journal_mode=WAL;")
|
||||||
conn.execute("PRAGMA synchronous=NORMAL;")
|
conn.execute("PRAGMA synchronous=NORMAL;")
|
||||||
|
|
||||||
conn.execute("""
|
conn.execute("""
|
||||||
CREATE TABLE IF NOT EXISTS main_page (
|
CREATE TABLE IF NOT EXISTS main_page (
|
||||||
url TEXT PRIMARY KEY,
|
url TEXT PRIMARY KEY,
|
||||||
@@ -18,7 +102,6 @@ def init_db() -> None:
|
|||||||
fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
)
|
)
|
||||||
""")
|
""")
|
||||||
|
|
||||||
conn.execute("""
|
conn.execute("""
|
||||||
CREATE TABLE IF NOT EXISTS chapter_page (
|
CREATE TABLE IF NOT EXISTS chapter_page (
|
||||||
url TEXT PRIMARY KEY,
|
url TEXT PRIMARY KEY,
|
||||||
|
|||||||
@@ -0,0 +1,251 @@
|
|||||||
|
import os
|
||||||
|
import re
|
||||||
|
from io import BytesIO
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from ebooklib import epub
|
||||||
|
from PIL import Image
|
||||||
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
from .database import db_fetch_one
|
||||||
|
from .parse import robust_fetch_bytes
|
||||||
|
|
||||||
|
|
||||||
|
_STYLE = """
|
||||||
|
@namespace epub "http://www.idpf.org/2007/ops";
|
||||||
|
body {
|
||||||
|
font-family: sans-serif;
|
||||||
|
font-size: 100%;
|
||||||
|
text-align: left;
|
||||||
|
padding-left: 5%;
|
||||||
|
padding-right: 5%;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
p {
|
||||||
|
margin-top: 0;
|
||||||
|
margin-bottom: 1em;
|
||||||
|
text-indent: 1.5em;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
text-align: center;
|
||||||
|
text-transform: uppercase;
|
||||||
|
margin-bottom: 2em;
|
||||||
|
}
|
||||||
|
img {
|
||||||
|
max-width: 100%;
|
||||||
|
height: auto;
|
||||||
|
display: block;
|
||||||
|
margin: 1em auto;
|
||||||
|
}
|
||||||
|
hr {
|
||||||
|
border: 0;
|
||||||
|
border-top: 1px solid #ccc;
|
||||||
|
margin: 2em 0;
|
||||||
|
}
|
||||||
|
aside.tl-note {
|
||||||
|
font-size: 90%;
|
||||||
|
margin: 1em 2em;
|
||||||
|
padding: 0.5em 1em;
|
||||||
|
border-left: 3px solid #ccc;
|
||||||
|
color: #555;
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _slugify(text: str) -> str:
|
||||||
|
slug = re.sub(r"[^\w\s-]", "", text.lower())
|
||||||
|
slug = re.sub(r"[-\s]+", "-", slug).strip("-_")
|
||||||
|
return slug or "novel"
|
||||||
|
|
||||||
|
|
||||||
|
def _get_clean_html(url: str) -> str:
|
||||||
|
row = db_fetch_one("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
|
||||||
|
if row is None or row[0] is None:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"No cleaned HTML cached for {url}. Run clean_chapters first."
|
||||||
|
)
|
||||||
|
return str(row[0])
|
||||||
|
|
||||||
|
|
||||||
|
def _download(url: str, referer: str | None = None) -> bytes:
|
||||||
|
status_code, content = robust_fetch_bytes(url, referer=referer)
|
||||||
|
if status_code != 200:
|
||||||
|
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
|
||||||
|
return content
|
||||||
|
|
||||||
|
|
||||||
|
def _compress_image(data: bytes) -> tuple[bytes, str]:
|
||||||
|
with Image.open(BytesIO(data)) as image:
|
||||||
|
image.load()
|
||||||
|
fmt = (image.format or "").upper()
|
||||||
|
|
||||||
|
buffer = BytesIO()
|
||||||
|
if fmt in ("JPEG", "JPG"):
|
||||||
|
image.save(buffer, format="JPEG", quality="keep")
|
||||||
|
return buffer.getvalue(), "jpg"
|
||||||
|
if fmt == "WEBP":
|
||||||
|
image.save(buffer, format="WEBP", quality=85)
|
||||||
|
return buffer.getvalue(), "webp"
|
||||||
|
if fmt == "PNG":
|
||||||
|
image.save(buffer, format="PNG", optimize=True)
|
||||||
|
return buffer.getvalue(), "png"
|
||||||
|
if fmt == "GIF":
|
||||||
|
image.save(buffer, format="GIF", optimize=True)
|
||||||
|
return buffer.getvalue(), "gif"
|
||||||
|
|
||||||
|
image.save(buffer, format="PNG", optimize=True)
|
||||||
|
return buffer.getvalue(), "png"
|
||||||
|
|
||||||
|
|
||||||
|
def _embed_images(
|
||||||
|
book: epub.EpubBook, html: str, image_cache: dict, referer: str | None = None
|
||||||
|
) -> str:
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
|
||||||
|
for img_tag in soup.find_all("img"):
|
||||||
|
src = img_tag.get("src")
|
||||||
|
if not src:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if src in image_cache:
|
||||||
|
img_tag["src"] = image_cache[src]
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
raw = _download(src, referer=referer)
|
||||||
|
content, ext = _compress_image(raw)
|
||||||
|
except Exception as e:
|
||||||
|
tqdm.write(f"Error downloading/compressing image {src}: {e}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
image_id = f"img_{len(image_cache) + 1}"
|
||||||
|
file_name = f"images/{image_id}.{ext}"
|
||||||
|
|
||||||
|
epub_img = epub.EpubImage()
|
||||||
|
epub_img.uid = image_id
|
||||||
|
epub_img.file_name = file_name
|
||||||
|
epub_img.content = content
|
||||||
|
book.add_item(epub_img)
|
||||||
|
|
||||||
|
img_tag["src"] = file_name
|
||||||
|
image_cache[src] = file_name
|
||||||
|
|
||||||
|
body = soup.body if soup.body is not None else soup
|
||||||
|
return body.decode_contents()
|
||||||
|
|
||||||
|
|
||||||
|
def create_epub(
|
||||||
|
metadata: dict,
|
||||||
|
output_dir: str = "output",
|
||||||
|
progress_callback=None,
|
||||||
|
cancel_event=None,
|
||||||
|
) -> str:
|
||||||
|
series = metadata.get("metadata", {})
|
||||||
|
title = series.get("title") or "Untitled"
|
||||||
|
author = series.get("author") or "Unknown"
|
||||||
|
language = series.get("language") or "en"
|
||||||
|
cover_url = series.get("cover_url")
|
||||||
|
|
||||||
|
book = epub.EpubBook()
|
||||||
|
book.set_identifier(_slugify(title))
|
||||||
|
book.set_title(title)
|
||||||
|
book.set_language(language)
|
||||||
|
book.add_author(author)
|
||||||
|
|
||||||
|
if cover_url:
|
||||||
|
try:
|
||||||
|
cover_raw = _download(cover_url, referer=metadata.get("novel_url"))
|
||||||
|
cover_bytes, cover_ext = _compress_image(cover_raw)
|
||||||
|
book.set_cover(f"cover.{cover_ext}", cover_bytes)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Cover download failed ({cover_url}): {e}. Proceeding without it.")
|
||||||
|
|
||||||
|
nav_css = epub.EpubItem(
|
||||||
|
uid="style_nav",
|
||||||
|
file_name="style/nav.css",
|
||||||
|
media_type="text/css",
|
||||||
|
content=_STYLE,
|
||||||
|
)
|
||||||
|
book.add_item(nav_css)
|
||||||
|
|
||||||
|
image_cache = {}
|
||||||
|
toc_structure = []
|
||||||
|
spine_structure = ["nav"]
|
||||||
|
chapter_counter = 1
|
||||||
|
|
||||||
|
volumes = metadata.get("volumes", [])
|
||||||
|
total_chapters = sum(len(v.get("chapters", [])) for v in volumes)
|
||||||
|
|
||||||
|
def _process_volume(volume, pbar=None):
|
||||||
|
nonlocal chapter_counter
|
||||||
|
volume_name = volume.get("name", "Volume")
|
||||||
|
sections = []
|
||||||
|
|
||||||
|
for chapter in volume.get("chapters", []):
|
||||||
|
if cancel_event is not None and cancel_event.is_set():
|
||||||
|
raise InterruptedError("EPUB compilation was cancelled by user")
|
||||||
|
|
||||||
|
chapter_name = chapter.get("name") or "Untitled"
|
||||||
|
url = chapter.get("url")
|
||||||
|
|
||||||
|
if pbar is not None:
|
||||||
|
pbar.set_postfix_str(chapter_name[:30])
|
||||||
|
pbar.update(1)
|
||||||
|
|
||||||
|
if progress_callback is not None:
|
||||||
|
progress_callback(stage="epub", current=chapter_counter, total=total_chapters, message=chapter_name)
|
||||||
|
|
||||||
|
if not url:
|
||||||
|
tqdm.write(f"[SKIP] No URL for chapter: {chapter_name}")
|
||||||
|
chapter_counter += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
clean_html = _get_clean_html(url)
|
||||||
|
except RuntimeError as e:
|
||||||
|
tqdm.write(f"[SKIP] {chapter_name}: {e}")
|
||||||
|
chapter_counter += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
clean_html = _embed_images(book, clean_html, image_cache, referer=url)
|
||||||
|
|
||||||
|
file_name = f"chapter_{chapter_counter}.xhtml"
|
||||||
|
epub_chapter = epub.EpubHtml(
|
||||||
|
title=chapter_name,
|
||||||
|
file_name=file_name,
|
||||||
|
lang=language,
|
||||||
|
)
|
||||||
|
epub_chapter.add_item(nav_css)
|
||||||
|
epub_chapter.content = f"<h1>{chapter_name}</h1>\n{clean_html}"
|
||||||
|
|
||||||
|
book.add_item(epub_chapter)
|
||||||
|
sections.append(epub_chapter)
|
||||||
|
spine_structure.append(epub_chapter)
|
||||||
|
|
||||||
|
chapter_counter += 1
|
||||||
|
|
||||||
|
if sections:
|
||||||
|
toc_structure.append((epub.Section(volume_name), tuple(sections)))
|
||||||
|
|
||||||
|
if progress_callback is None:
|
||||||
|
with tqdm(total=total_chapters, desc="Building EPUB", unit="ch") as pbar:
|
||||||
|
for volume in volumes:
|
||||||
|
_process_volume(volume, pbar=pbar)
|
||||||
|
else:
|
||||||
|
for volume in volumes:
|
||||||
|
_process_volume(volume, pbar=None)
|
||||||
|
|
||||||
|
if not spine_structure[1:]:
|
||||||
|
raise RuntimeError("No chapters were added to the EPUB.")
|
||||||
|
|
||||||
|
book.toc = toc_structure
|
||||||
|
book.add_item(epub.EpubNav())
|
||||||
|
book.add_item(epub.EpubNcx())
|
||||||
|
book.spine = spine_structure
|
||||||
|
|
||||||
|
os.makedirs(output_dir, exist_ok=True)
|
||||||
|
output_path = os.path.join(output_dir, f"{_slugify(title)}.epub")
|
||||||
|
|
||||||
|
epub.write_epub(output_path, book, {})
|
||||||
|
print(f"Successfully generated: {output_path}")
|
||||||
|
return output_path
|
||||||
|
|||||||
+51
@@ -0,0 +1,51 @@
|
|||||||
|
import os
|
||||||
|
from openai import OpenAI
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
MODEL = os.environ.get("LLM_MODEL")
|
||||||
|
if not MODEL:
|
||||||
|
raise RuntimeError("LLM_MODEL is not set. Add it to your .env file.")
|
||||||
|
|
||||||
|
BASE_URL = os.environ.get("LLM_BASE_URL")
|
||||||
|
if not BASE_URL:
|
||||||
|
raise RuntimeError("LLM_BASE_URL is not set. Add it to your .env file.")
|
||||||
|
|
||||||
|
MAX_TOKENS = int(os.environ.get("LLM_MAX_TOKENS", "65536"))
|
||||||
|
|
||||||
|
API_KEY = os.environ.get("LLM_API_KEY")
|
||||||
|
if not API_KEY:
|
||||||
|
raise RuntimeError("LLM_API_KEY is not set. Add it to your .env file.")
|
||||||
|
|
||||||
|
client = OpenAI(
|
||||||
|
api_key=API_KEY,
|
||||||
|
base_url=BASE_URL,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _request(system_prompt: str, user_content: str, response_format=None) -> str:
|
||||||
|
kwargs = {
|
||||||
|
"model": MODEL,
|
||||||
|
"messages": [
|
||||||
|
{"role": "system", "content": system_prompt},
|
||||||
|
{"role": "user", "content": user_content},
|
||||||
|
],
|
||||||
|
"temperature": 0.0,
|
||||||
|
"max_tokens": MAX_TOKENS,
|
||||||
|
"extra_body": {"thinking": {"type": "disabled"}},
|
||||||
|
}
|
||||||
|
if response_format is not None:
|
||||||
|
kwargs["response_format"] = response_format
|
||||||
|
|
||||||
|
response = client.chat.completions.create(**kwargs)
|
||||||
|
return (response.choices[0].message.content or "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def generate_json(system_prompt: str, user_content: str) -> str:
|
||||||
|
return _request(system_prompt, user_content, response_format={"type": "json_object"})
|
||||||
|
|
||||||
|
|
||||||
|
def generate_text(system_prompt: str, user_content: str) -> str:
|
||||||
|
return _request(system_prompt, user_content)
|
||||||
+16
-34
@@ -1,57 +1,39 @@
|
|||||||
import json
|
import json
|
||||||
import os
|
from .database import db_execute, db_fetch_one
|
||||||
import re
|
|
||||||
import sqlite3
|
|
||||||
from google import genai
|
|
||||||
from google.genai import types
|
|
||||||
from dotenv import load_dotenv
|
|
||||||
from .database import DB_PATH
|
|
||||||
from .parse import cache_hit
|
from .parse import cache_hit
|
||||||
from .clean_html import clean_novel_html
|
from .clean_html import clean_novel_html
|
||||||
from .prompt import INDEX_EXTRACTION_SYSTEM_PROMPT
|
from .prompt import INDEX_EXTRACTION_SYSTEM_PROMPT
|
||||||
|
from .llm import generate_json
|
||||||
|
|
||||||
|
|
||||||
load_dotenv()
|
def extract_metadata_json(
|
||||||
client = genai.Client()
|
html_content: str, url: str, force_refresh: bool = False
|
||||||
|
) -> dict:
|
||||||
|
if not force_refresh:
|
||||||
def extract_metadata_json(html_content: str, url: str) -> dict:
|
row = db_fetch_one("SELECT json_text FROM main_page WHERE url = ?", (url,))
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
|
||||||
cursor = conn.cursor()
|
|
||||||
cursor.execute("SELECT json_text FROM main_page WHERE url = ?", (url,))
|
|
||||||
row = cursor.fetchone()
|
|
||||||
if row is not None and row[0] is not None:
|
if row is not None and row[0] is not None:
|
||||||
print("Metadata JSON Cache hit!")
|
|
||||||
return json.loads(row[0])
|
return json.loads(row[0])
|
||||||
|
|
||||||
response = client.models.generate_content(
|
raw_output = generate_json(INDEX_EXTRACTION_SYSTEM_PROMPT, html_content)
|
||||||
model="gemini-3.6-flash",
|
|
||||||
contents=html_content,
|
|
||||||
config=types.GenerateContentConfig(
|
|
||||||
system_instruction=INDEX_EXTRACTION_SYSTEM_PROMPT,
|
|
||||||
temperature=0.0,
|
|
||||||
response_mime_type="application/json",
|
|
||||||
tools=None
|
|
||||||
),
|
|
||||||
)
|
|
||||||
raw_output = response.text.strip()
|
|
||||||
|
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
db_execute("""
|
||||||
conn.execute("""
|
|
||||||
INSERT INTO main_page (url, json_text)
|
INSERT INTO main_page (url, json_text)
|
||||||
VALUES (?, ?)
|
VALUES (?, ?)
|
||||||
ON CONFLICT(url) DO UPDATE SET json_text = excluded.json_text
|
ON CONFLICT(url) DO UPDATE SET json_text = excluded.json_text
|
||||||
""", (url, raw_output))
|
""", (url, raw_output))
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
return json.loads(raw_output)
|
return json.loads(raw_output)
|
||||||
|
|
||||||
|
|
||||||
def write_json(page_url: str, output_json_path: str) -> None:
|
def write_json(
|
||||||
html_content = clean_novel_html(cache_hit(page_url))
|
page_url: str, output_json_path: str, force_refresh: bool = False
|
||||||
|
) -> None:
|
||||||
|
html_content = clean_novel_html(cache_hit(page_url, force_refresh=force_refresh))
|
||||||
print("Extracting metadata via LLM...")
|
print("Extracting metadata via LLM...")
|
||||||
|
|
||||||
extracted_data = extract_metadata_json(html_content, page_url)
|
extracted_data = extract_metadata_json(
|
||||||
|
html_content, page_url, force_refresh=force_refresh
|
||||||
|
)
|
||||||
with open(output_json_path, "w", encoding="utf-8") as f:
|
with open(output_json_path, "w", encoding="utf-8") as f:
|
||||||
json.dump(extracted_data, f, indent=2, ensure_ascii=False)
|
json.dump(extracted_data, f, indent=2, ensure_ascii=False)
|
||||||
print(f"\n[SUCCESS] Extracted JSON saved to: {output_json_path}\n")
|
print(f"\n[SUCCESS] Extracted JSON saved to: {output_json_path}\n")
|
||||||
|
|||||||
+94
-21
@@ -1,17 +1,22 @@
|
|||||||
import random
|
import random
|
||||||
|
import re
|
||||||
import time
|
import time
|
||||||
import sqlite3
|
from urllib.parse import urljoin
|
||||||
from curl_cffi import requests
|
from curl_cffi import requests
|
||||||
from curl_cffi.requests.errors import RequestsError
|
from curl_cffi.requests.errors import RequestsError
|
||||||
from .database import DB_PATH
|
from .database import db_execute, db_fetch_one
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
session = requests.Session(impersonate="chrome120")
|
session = requests.Session(impersonate="chrome120")
|
||||||
|
|
||||||
def robust_fetch(
|
|
||||||
url: str, min_delay: float = 2.0, max_delay: float = 5.0, retries: int = 3
|
def _fetch_response(
|
||||||
) -> str:
|
url: str,
|
||||||
|
min_delay: float,
|
||||||
|
max_delay: float,
|
||||||
|
retries: int,
|
||||||
|
headers: dict | None = None,
|
||||||
|
):
|
||||||
sleep_time = random.uniform(min_delay, max_delay)
|
sleep_time = random.uniform(min_delay, max_delay)
|
||||||
if random.random() < 0.08:
|
if random.random() < 0.08:
|
||||||
sleep_time += random.uniform(4.0, 8.0)
|
sleep_time += random.uniform(4.0, 8.0)
|
||||||
@@ -19,7 +24,7 @@ def robust_fetch(
|
|||||||
|
|
||||||
for attempt in range(retries):
|
for attempt in range(retries):
|
||||||
try:
|
try:
|
||||||
response = session.get(url, timeout=15)
|
response = session.get(url, headers=headers, timeout=15)
|
||||||
|
|
||||||
if response.status_code in (429, 503):
|
if response.status_code in (429, 503):
|
||||||
retry_after = response.headers.get("Retry-After")
|
retry_after = response.headers.get("Retry-After")
|
||||||
@@ -36,7 +41,7 @@ def robust_fetch(
|
|||||||
"Cloudflare Managed Challenge encountered. Headless browser required."
|
"Cloudflare Managed Challenge encountered. Headless browser required."
|
||||||
)
|
)
|
||||||
|
|
||||||
return response.status_code, str(response.text)
|
return response
|
||||||
|
|
||||||
except (RequestsError, TimeoutError) as e:
|
except (RequestsError, TimeoutError) as e:
|
||||||
if attempt == retries - 1:
|
if attempt == retries - 1:
|
||||||
@@ -48,25 +53,93 @@ def robust_fetch(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def cache_hit(url: str) -> str:
|
def robust_fetch(
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
url: str,
|
||||||
cursor = conn.cursor()
|
min_delay: float = 2.0,
|
||||||
cursor.execute("SELECT html_text FROM main_page WHERE url = ?", (url,))
|
max_delay: float = 5.0,
|
||||||
row = cursor.fetchone()
|
retries: int = 3,
|
||||||
if row is not None:
|
headers: dict | None = None,
|
||||||
print("Request cache hit!")
|
) -> tuple[int, str]:
|
||||||
|
response = _fetch_response(url, min_delay, max_delay, retries, headers=headers)
|
||||||
|
return response.status_code, str(response.text)
|
||||||
|
|
||||||
|
|
||||||
|
def robust_fetch_bytes(
|
||||||
|
url: str,
|
||||||
|
min_delay: float = 2.0,
|
||||||
|
max_delay: float = 5.0,
|
||||||
|
retries: int = 3,
|
||||||
|
headers: dict | None = None,
|
||||||
|
referer: str | None = None,
|
||||||
|
) -> tuple[int, bytes]:
|
||||||
|
req_headers = {
|
||||||
|
"Accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
|
||||||
|
"Sec-Fetch-Dest": "image",
|
||||||
|
"Sec-Fetch-Mode": "no-cors",
|
||||||
|
"Sec-Fetch-Site": "cross-site",
|
||||||
|
}
|
||||||
|
if referer:
|
||||||
|
req_headers["Referer"] = referer
|
||||||
|
if headers:
|
||||||
|
req_headers.update(headers)
|
||||||
|
|
||||||
|
response = _fetch_response(
|
||||||
|
url, min_delay, max_delay, retries, headers=req_headers
|
||||||
|
)
|
||||||
|
return response.status_code, response.content
|
||||||
|
|
||||||
|
# Exclusive to ZTL
|
||||||
|
def _fetch_wp_manga_chapters(base_url: str, html: str) -> str:
|
||||||
|
manga_id_match = (
|
||||||
|
re.search(r'["\']manga_id["\']\s*:\s*["\']?(\d+)', html)
|
||||||
|
or re.search(r'["\']wc_post_id["\']\s*:\s*["\']?(\d+)', html)
|
||||||
|
or re.search(r'postid-(\d+)', html)
|
||||||
|
)
|
||||||
|
if not manga_id_match:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
manga_id = manga_id_match.group(1)
|
||||||
|
|
||||||
|
ajax_url_match = re.search(r'["\']ajax_url["\']\s*:\s*["\'](http[^"\']+)', html)
|
||||||
|
if ajax_url_match:
|
||||||
|
ajax_url = ajax_url_match.group(1).replace(r"\/", "/")
|
||||||
|
else:
|
||||||
|
ajax_url = urljoin(base_url, "/wp-admin/admin-ajax.php")
|
||||||
|
|
||||||
|
try:
|
||||||
|
res = session.post(
|
||||||
|
ajax_url,
|
||||||
|
data={"action": "manga_get_chapters", "manga": manga_id},
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
if res.status_code == 200:
|
||||||
|
return res.text
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def cache_hit(url: str, force_refresh: bool = False) -> str:
|
||||||
|
if not force_refresh:
|
||||||
|
row = db_fetch_one("SELECT html_text FROM main_page WHERE url = ?", (url,))
|
||||||
|
if row is not None and row[0] is not None:
|
||||||
return str(row[0])
|
return str(row[0])
|
||||||
|
|
||||||
# Cache miss: fetch and save
|
|
||||||
status_code, html = robust_fetch(url)
|
status_code, html = robust_fetch(url)
|
||||||
if status_code != 200:
|
if status_code != 200:
|
||||||
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
|
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
|
||||||
|
|
||||||
with sqlite3.connect(DB_PATH) as conn:
|
extra_chapters = _fetch_wp_manga_chapters(url, html)
|
||||||
conn.execute(
|
if extra_chapters:
|
||||||
"INSERT OR REPLACE INTO main_page (url, status_code, html_text) VALUES (?, ?, ?)",
|
if "</body>" in html:
|
||||||
(url, status_code, html)
|
html = html.replace("</body>", f"<div class='injected-wp-manga-chapters'>{extra_chapters}</div></body>")
|
||||||
|
else:
|
||||||
|
html = f"{html}\n<div class='injected-wp-manga-chapters'>{extra_chapters}</div>"
|
||||||
|
|
||||||
|
db_execute(
|
||||||
|
"INSERT INTO main_page (url, status_code, html_text) VALUES (?, ?, ?) ON CONFLICT(url) DO UPDATE SET status_code=excluded.status_code, html_text=excluded.html_text, fetched_at=CURRENT_TIMESTAMP",
|
||||||
|
(url, status_code, html),
|
||||||
)
|
)
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
return html
|
return html
|
||||||
+181
@@ -0,0 +1,181 @@
|
|||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import FastAPI, HTTPException, Request
|
||||||
|
from fastapi.responses import HTMLResponse, FileResponse, StreamingResponse
|
||||||
|
from fastapi.staticfiles import StaticFiles
|
||||||
|
from fastapi.templating import Jinja2Templates
|
||||||
|
|
||||||
|
from .job_manager import (
|
||||||
|
jobs,
|
||||||
|
job_queues,
|
||||||
|
create_job,
|
||||||
|
cancel_job,
|
||||||
|
cancel_all_jobs,
|
||||||
|
run_job,
|
||||||
|
)
|
||||||
|
|
||||||
|
OUTPUT_DIR = Path(os.environ.get("OUTPUT_DIR", "output")).resolve()
|
||||||
|
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
yield
|
||||||
|
cancel_all_jobs()
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(title="Silksoup", lifespan=lifespan)
|
||||||
|
|
||||||
|
BASE_DIR = Path(__file__).resolve().parent
|
||||||
|
templates = Jinja2Templates(directory=str(BASE_DIR / "templates"))
|
||||||
|
|
||||||
|
if (BASE_DIR / "static").exists():
|
||||||
|
app.mount("/static", StaticFiles(directory=str(BASE_DIR / "static")), name="static")
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/", response_class=HTMLResponse)
|
||||||
|
async def index(request: Request):
|
||||||
|
return templates.TemplateResponse(request, "index.html")
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/jobs")
|
||||||
|
async def start_job(payload: dict):
|
||||||
|
url = (payload.get("url") or "").strip()
|
||||||
|
if not url or not (url.startswith("http://") or url.startswith("https://")):
|
||||||
|
raise HTTPException(status_code=400, detail="Valid URL required")
|
||||||
|
force_refresh = bool(payload.get("force_refresh", False))
|
||||||
|
job_id = create_job(url, force_refresh=force_refresh, output_dir=str(OUTPUT_DIR))
|
||||||
|
asyncio.create_task(run_job(job_id))
|
||||||
|
return {"job_id": job_id, "status": "queued"}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/jobs/{job_id}/cancel")
|
||||||
|
async def stop_job(job_id: str):
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
|
if job.get("status") not in ("running", "queued"):
|
||||||
|
return {"status": job.get("status"), "message": "Job is not running"}
|
||||||
|
cancel_job(job_id)
|
||||||
|
return {"status": "cancelled", "message": "Job cancelled successfully"}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/jobs")
|
||||||
|
async def list_jobs():
|
||||||
|
items = []
|
||||||
|
for job_id, job in sorted(jobs.items(), key=lambda item: item[1].get("created_at", 0), reverse=True):
|
||||||
|
items.append({
|
||||||
|
"id": job["id"],
|
||||||
|
"url": job["url"],
|
||||||
|
"status": job["status"],
|
||||||
|
"stage": job["stage"],
|
||||||
|
"current": job["current"],
|
||||||
|
"total": job["total"],
|
||||||
|
"message": job["message"],
|
||||||
|
"created_at": job.get("created_at"),
|
||||||
|
"result_path": job.get("result_path"),
|
||||||
|
"error": job.get("error"),
|
||||||
|
})
|
||||||
|
return {"jobs": items[:20]}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/jobs/{job_id}")
|
||||||
|
async def get_job(job_id: str):
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
|
return {
|
||||||
|
"id": job["id"],
|
||||||
|
"url": job["url"],
|
||||||
|
"status": job["status"],
|
||||||
|
"stage": job["stage"],
|
||||||
|
"current": job["current"],
|
||||||
|
"total": job["total"],
|
||||||
|
"message": job["message"],
|
||||||
|
"error": job["error"],
|
||||||
|
"result_path": job["result_path"],
|
||||||
|
"logs": job["logs"][-200:],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/jobs/{job_id}/events")
|
||||||
|
async def job_events(job_id: str):
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
raise HTTPException(status_code=404, detail="Job not found")
|
||||||
|
|
||||||
|
async def event_gen():
|
||||||
|
q = job_queues.get(job_id)
|
||||||
|
if q is None:
|
||||||
|
return
|
||||||
|
last_idx = 0
|
||||||
|
events = job.get("events", [])
|
||||||
|
for evt in events[last_idx:]:
|
||||||
|
yield f"data: {json.dumps(evt)}\n\n"
|
||||||
|
last_idx += 1
|
||||||
|
if evt.get("message") == "__DONE__":
|
||||||
|
return
|
||||||
|
while True:
|
||||||
|
if job["status"] in ("completed", "failed", "cancelled") and q.empty():
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
evt = await asyncio.wait_for(q.get(), timeout=15.0)
|
||||||
|
yield f"data: {json.dumps(evt)}\n\n"
|
||||||
|
if evt.get("message") == "__DONE__":
|
||||||
|
break
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
yield ": keepalive\n\n"
|
||||||
|
if job["status"] in ("completed", "failed", "cancelled"):
|
||||||
|
break
|
||||||
|
|
||||||
|
return StreamingResponse(event_gen(), media_type="text/event-stream", headers={
|
||||||
|
"Cache-Control": "no-cache",
|
||||||
|
"Connection": "keep-alive",
|
||||||
|
"X-Accel-Buffering": "no",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/epubs")
|
||||||
|
async def list_epubs():
|
||||||
|
files = []
|
||||||
|
if OUTPUT_DIR.exists():
|
||||||
|
for p in sorted(OUTPUT_DIR.glob("*.epub"), key=lambda x: x.stat().st_mtime, reverse=True):
|
||||||
|
stat = p.stat()
|
||||||
|
files.append({
|
||||||
|
"name": p.name,
|
||||||
|
"size": stat.st_size,
|
||||||
|
"size_human": _human_size(stat.st_size),
|
||||||
|
"mtime": stat.st_mtime,
|
||||||
|
})
|
||||||
|
return {"epubs": files, "output_dir": str(OUTPUT_DIR)}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/epubs/{filename}")
|
||||||
|
async def download_epub(filename: str):
|
||||||
|
safe = Path(filename).name
|
||||||
|
target = OUTPUT_DIR / safe
|
||||||
|
if not target.exists() or not target.is_file():
|
||||||
|
raise HTTPException(status_code=404, detail="File not found")
|
||||||
|
return FileResponse(str(target), media_type="application/epub+zip", filename=safe)
|
||||||
|
|
||||||
|
|
||||||
|
@app.delete("/api/epubs/{filename}")
|
||||||
|
async def delete_epub(filename: str):
|
||||||
|
safe = Path(filename).name
|
||||||
|
target = OUTPUT_DIR / safe
|
||||||
|
if not target.exists():
|
||||||
|
raise HTTPException(status_code=404, detail="File not found")
|
||||||
|
target.unlink()
|
||||||
|
return {"deleted": safe}
|
||||||
|
|
||||||
|
|
||||||
|
def _human_size(n: int) -> str:
|
||||||
|
for unit in ["B", "KB", "MB", "GB"]:
|
||||||
|
if n < 1024:
|
||||||
|
return f"{n:.1f} {unit}" if unit != "B" else f"{n} B"
|
||||||
|
n /= 1024
|
||||||
|
return f"{n:.1f} TB"
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
import asyncio
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from src.database import init_db
|
||||||
|
from src.parse import cache_hit
|
||||||
|
from src.clean_html import clean_novel_html
|
||||||
|
from src.metadata import extract_metadata_json
|
||||||
|
from src.chapter import clean_chapters
|
||||||
|
from src.epub_creator import create_epub
|
||||||
|
|
||||||
|
jobs: dict[str, dict] = {}
|
||||||
|
job_queues: dict[str, asyncio.Queue] = {}
|
||||||
|
cancel_events: dict[str, threading.Event] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def _push(job_id: str, event: dict):
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if job is None:
|
||||||
|
return
|
||||||
|
job["events"].append(event)
|
||||||
|
job["updated_at"] = time.time()
|
||||||
|
if "stage" in event:
|
||||||
|
job["stage"] = event["stage"]
|
||||||
|
if "current" in event:
|
||||||
|
job["current"] = event["current"]
|
||||||
|
if "total" in event:
|
||||||
|
job["total"] = event["total"]
|
||||||
|
if "message" in event:
|
||||||
|
job["message"] = event["message"]
|
||||||
|
q = job_queues.get(job_id)
|
||||||
|
if q is not None:
|
||||||
|
try:
|
||||||
|
q.put_nowait(event)
|
||||||
|
except asyncio.QueueFull:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def create_job(url: str, force_refresh: bool = False, output_dir: str = "output") -> str:
|
||||||
|
job_id = uuid.uuid4().hex[:12]
|
||||||
|
jobs[job_id] = {
|
||||||
|
"id": job_id,
|
||||||
|
"url": url,
|
||||||
|
"force_refresh": force_refresh,
|
||||||
|
"output_dir": output_dir,
|
||||||
|
"status": "queued",
|
||||||
|
"stage": "queued",
|
||||||
|
"current": 0,
|
||||||
|
"total": 0,
|
||||||
|
"message": "Queued",
|
||||||
|
"logs": [],
|
||||||
|
"events": [],
|
||||||
|
"result_path": None,
|
||||||
|
"error": None,
|
||||||
|
"created_at": time.time(),
|
||||||
|
"updated_at": time.time(),
|
||||||
|
}
|
||||||
|
job_queues[job_id] = asyncio.Queue(maxsize=1000)
|
||||||
|
cancel_events[job_id] = threading.Event()
|
||||||
|
return job_id
|
||||||
|
|
||||||
|
|
||||||
|
def cancel_job(job_id: str) -> bool:
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if not job:
|
||||||
|
return False
|
||||||
|
evt = cancel_events.get(job_id)
|
||||||
|
if evt is not None:
|
||||||
|
evt.set()
|
||||||
|
job["status"] = "cancelled"
|
||||||
|
job["stage"] = "cancelled"
|
||||||
|
job["message"] = "Process stopped by user"
|
||||||
|
job["logs"].append("[CANCEL] Process stopped by user. Cached chapters remain safe.")
|
||||||
|
_push(job_id, {"stage": "cancelled", "current": job.get("current", 0), "total": job.get("total", 0), "message": "Process stopped by user"})
|
||||||
|
_push(job_id, {"stage": "done", "current": job.get("current", 0), "total": job.get("total", 0), "message": "__DONE__"})
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def cancel_all_jobs():
|
||||||
|
for job_id, evt in cancel_events.items():
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if job and job.get("status") in ("running", "queued"):
|
||||||
|
evt.set()
|
||||||
|
job["status"] = "cancelled"
|
||||||
|
job["stage"] = "cancelled"
|
||||||
|
|
||||||
|
|
||||||
|
async def run_job(job_id: str):
|
||||||
|
job = jobs.get(job_id)
|
||||||
|
if job is None:
|
||||||
|
return
|
||||||
|
url = job["url"]
|
||||||
|
force_refresh = job["force_refresh"]
|
||||||
|
output_dir = job["output_dir"]
|
||||||
|
cancel_evt = cancel_events.get(job_id)
|
||||||
|
job["status"] = "running"
|
||||||
|
|
||||||
|
def _cb(stage=None, current=None, total=None, message=None, **kwargs):
|
||||||
|
evt = {"stage": stage or "unknown", "current": current or 0, "total": total or 0, "message": message or ""}
|
||||||
|
_push(job_id, evt)
|
||||||
|
if message:
|
||||||
|
job["logs"].append(f"[{stage}] {message}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
if cancel_evt and cancel_evt.is_set():
|
||||||
|
raise InterruptedError("Cancelled before start")
|
||||||
|
|
||||||
|
init_db()
|
||||||
|
_push(job_id, {"stage": "fetching", "current": 0, "total": 1, "message": "Fetching index page..."})
|
||||||
|
job["logs"].append(f"Fetching {url}")
|
||||||
|
|
||||||
|
html = await asyncio.to_thread(cache_hit, url, force_refresh)
|
||||||
|
if cancel_evt and cancel_evt.is_set():
|
||||||
|
raise InterruptedError("Cancelled after fetch")
|
||||||
|
|
||||||
|
cleaned = clean_novel_html(html)
|
||||||
|
|
||||||
|
_push(job_id, {"stage": "metadata", "current": 0, "total": 1, "message": "Extracting metadata via LLM..."})
|
||||||
|
job["logs"].append("Extracting metadata")
|
||||||
|
|
||||||
|
metadata = await asyncio.to_thread(extract_metadata_json, cleaned, url, force_refresh)
|
||||||
|
metadata["novel_url"] = url
|
||||||
|
|
||||||
|
if cancel_evt and cancel_evt.is_set():
|
||||||
|
raise InterruptedError("Cancelled after metadata")
|
||||||
|
|
||||||
|
volumes = metadata.get("volumes", [])
|
||||||
|
total_chapters = sum(len(v.get("chapters", [])) for v in volumes)
|
||||||
|
_push(job_id, {"stage": "cleaning", "current": 0, "total": total_chapters, "message": f"Cleaning {total_chapters} chapters"})
|
||||||
|
job["logs"].append(f"Found {len(volumes)} volumes, {total_chapters} chapters")
|
||||||
|
|
||||||
|
await asyncio.to_thread(clean_chapters, metadata, force_refresh, _cb, cancel_evt)
|
||||||
|
|
||||||
|
if cancel_evt and cancel_evt.is_set():
|
||||||
|
raise InterruptedError("Cancelled before EPUB building")
|
||||||
|
|
||||||
|
_push(job_id, {"stage": "epub", "current": 0, "total": total_chapters, "message": "Building EPUB..."})
|
||||||
|
output_path = await asyncio.to_thread(create_epub, metadata, output_dir, _cb, cancel_evt)
|
||||||
|
|
||||||
|
job["result_path"] = output_path
|
||||||
|
job["status"] = "completed"
|
||||||
|
job["stage"] = "completed"
|
||||||
|
job["message"] = f"Completed: {Path(output_path).name}"
|
||||||
|
job["logs"].append(f"Generated {output_path}")
|
||||||
|
_push(job_id, {"stage": "completed", "current": total_chapters, "total": total_chapters, "message": job["message"]})
|
||||||
|
_push(job_id, {"stage": "done", "current": total_chapters, "total": total_chapters, "message": "__DONE__"})
|
||||||
|
except InterruptedError:
|
||||||
|
job["status"] = "cancelled"
|
||||||
|
job["stage"] = "cancelled"
|
||||||
|
job["message"] = "Process stopped by user"
|
||||||
|
job["logs"].append("[CANCEL] Process stopped by user. Cached chapters remain safe.")
|
||||||
|
_push(job_id, {"stage": "cancelled", "current": job.get("current", 0), "total": job.get("total", 0), "message": "Process stopped by user"})
|
||||||
|
_push(job_id, {"stage": "done", "current": job.get("current", 0), "total": job.get("total", 0), "message": "__DONE__"})
|
||||||
|
except Exception as e:
|
||||||
|
job["status"] = "failed"
|
||||||
|
job["stage"] = "failed"
|
||||||
|
job["error"] = str(e)
|
||||||
|
job["message"] = str(e)
|
||||||
|
job["logs"].append(f"[ERROR] {e}")
|
||||||
|
_push(job_id, {"stage": "failed", "current": job.get("current", 0), "total": job.get("total", 0), "message": str(e)})
|
||||||
|
_push(job_id, {"stage": "done", "current": 0, "total": 0, "message": "__DONE__"})
|
||||||
@@ -0,0 +1,541 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en" class="dark">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Silksoup — Novel to EPUB</title>
|
||||||
|
<script src="https://cdn.tailwindcss.com"></script>
|
||||||
|
<script>
|
||||||
|
tailwind.config = { darkMode: 'class', theme: { extend: { fontFamily: { sans: ['Inter','system-ui','sans-serif'] } } } }
|
||||||
|
</script>
|
||||||
|
<style>
|
||||||
|
::-webkit-scrollbar{width:6px;height:6px}::-webkit-scrollbar-thumb{background:#27272a;border-radius:999px}::-webkit-scrollbar-track{background:transparent}
|
||||||
|
@keyframes pulseGlow { 0%, 100% { opacity: 0.4; } 50% { opacity: 1; } }
|
||||||
|
.animate-glow { animation: pulseGlow 2s cubic-bezier(0.4, 0, 0.6, 1) infinite; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body class="bg-[#09090b] text-zinc-100 min-h-screen flex flex-col antialiased selection:bg-zinc-800">
|
||||||
|
<header class="border-b border-zinc-800/80 bg-zinc-950/80 backdrop-blur sticky top-0 z-30">
|
||||||
|
<div class="max-w-4xl mx-auto px-4 py-3.5 flex items-center justify-between">
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
|
<div class="w-8 h-8 rounded-lg bg-white text-black font-black grid place-items-center text-sm shadow-sm">S</div>
|
||||||
|
<div>
|
||||||
|
<h1 class="font-semibold tracking-tight text-sm leading-none flex items-center gap-2">
|
||||||
|
Silksoup
|
||||||
|
<span id="headerBadge" class="hidden text-[10px] font-medium px-2 py-0.5 rounded-full bg-emerald-500/10 text-emerald-400 border border-emerald-500/20">Active</span>
|
||||||
|
</h1>
|
||||||
|
<p class="text-[11px] text-zinc-400 mt-0.5">Novel to EPUB pipeline</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
|
<button id="notifyBtn" onclick="toggleNotifications()" title="Enable Desktop Notifications" class="text-xs text-zinc-400 hover:text-zinc-200 px-2.5 py-1.5 rounded-lg border border-zinc-800 hover:border-zinc-700 transition flex items-center gap-1.5">
|
||||||
|
<span id="notifyIcon">🔔</span>
|
||||||
|
<span id="notifyText" class="hidden sm:inline">Notify</span>
|
||||||
|
</button>
|
||||||
|
<a href="/api/epubs" target="_blank" class="text-xs text-zinc-400 hover:text-zinc-200 px-2.5 py-1.5 rounded-lg border border-zinc-800 hover:border-zinc-700 transition">API</a>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main class="max-w-4xl mx-auto w-full px-4 py-6 flex flex-col gap-6 flex-1">
|
||||||
|
<!-- New Job Card -->
|
||||||
|
<section class="rounded-2xl border border-zinc-800/80 bg-zinc-900/40 p-5 shadow-sm">
|
||||||
|
<div class="flex items-center justify-between mb-3">
|
||||||
|
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Scrape New Novel</h2>
|
||||||
|
<button onclick="pasteClipboard()" class="text-[11px] text-zinc-400 hover:text-white flex items-center gap-1 transition">
|
||||||
|
<span>📋</span> Paste URL
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<form id="jobForm" class="space-y-4" onsubmit="return false">
|
||||||
|
<div class="relative">
|
||||||
|
<input id="urlInput" type="url" required placeholder="https://kakuyomu.jp/works/... or novel index URL" class="w-full rounded-xl bg-zinc-950 border border-zinc-800 px-4 py-3.5 text-sm placeholder:text-zinc-600 focus:outline-none focus:border-zinc-500 transition shadow-inner">
|
||||||
|
</div>
|
||||||
|
<div class="flex flex-col sm:flex-row sm:items-center justify-between gap-3 pt-1">
|
||||||
|
<label class="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none">
|
||||||
|
<input id="refreshInput" type="checkbox" class="rounded border-zinc-700 bg-zinc-950 text-white focus:ring-0 focus:ring-offset-0">
|
||||||
|
Force refresh (bypass database cache)
|
||||||
|
</label>
|
||||||
|
<button id="startBtn" class="sm:w-auto w-full rounded-xl bg-white text-black text-sm font-semibold px-7 py-3 hover:bg-zinc-200 active:scale-[0.99] transition shrink-0 flex items-center justify-center gap-2">
|
||||||
|
<span id="btnSpinner" class="hidden inline-block w-4 h-4 border-2 border-black border-t-transparent rounded-full animate-spin"></span>
|
||||||
|
<span id="btnText">Start Scraping → EPUB</span>
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<!-- Active / In-Progress Job Section -->
|
||||||
|
<section id="activeJob" class="hidden rounded-2xl border border-zinc-800 bg-zinc-900/60 p-5 shadow-sm transition-all">
|
||||||
|
<div class="flex items-center justify-between mb-3">
|
||||||
|
<div class="flex items-center gap-2">
|
||||||
|
<span id="statusPulse" class="w-2.5 h-2.5 rounded-full bg-emerald-400 animate-glow"></span>
|
||||||
|
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Task Monitor</h2>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-2">
|
||||||
|
<button id="cancelJobBtn" onclick="requestCancelJob()" class="text-[11px] px-2.5 py-0.5 rounded-full border border-rose-900/60 bg-rose-950/40 text-rose-300 hover:bg-rose-900/60 transition">Stop Process</button>
|
||||||
|
<span id="jobStatusBadge" class="text-[11px] font-mono uppercase px-2.5 py-0.5 rounded-full bg-zinc-800 text-zinc-300 border border-zinc-700">idle</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="space-y-3.5">
|
||||||
|
<div class="flex items-center justify-between text-xs gap-3">
|
||||||
|
<div class="flex items-center gap-2 min-w-0">
|
||||||
|
<span id="jobStage" class="px-2 py-0.5 rounded bg-zinc-800/80 text-zinc-300 font-mono text-[10px] uppercase tracking-wider">init</span>
|
||||||
|
<span id="jobMessage" class="text-zinc-300 truncate font-medium">Starting...</span>
|
||||||
|
</div>
|
||||||
|
<span id="jobCounter" class="text-zinc-400 tabular-nums font-mono font-medium shrink-0">0 / 0</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="h-2 rounded-full bg-zinc-950 border border-zinc-800 overflow-hidden relative">
|
||||||
|
<div id="jobProgress" class="h-full bg-white transition-all duration-300" style="width:0%"></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Log Toolbar -->
|
||||||
|
<div class="flex items-center justify-between text-[11px] text-zinc-500 px-0.5 pt-1">
|
||||||
|
<span>Execution Log</span>
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
|
<label class="flex items-center gap-1.5 cursor-pointer select-none hover:text-zinc-300">
|
||||||
|
<input id="autoScrollCheck" type="checkbox" checked class="rounded border-zinc-700 bg-zinc-950 text-white focus:ring-0">
|
||||||
|
Auto-scroll
|
||||||
|
</label>
|
||||||
|
<button onclick="clearLogs()" class="hover:text-zinc-300">Clear</button>
|
||||||
|
<button onclick="toggleLogHeight()" id="expandLogBtn" class="hover:text-zinc-300">Expand</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="jobLog" class="h-44 overflow-auto rounded-xl bg-zinc-950 border border-zinc-800/90 p-3.5 font-mono text-[11px] leading-relaxed text-zinc-400 whitespace-pre-wrap transition-all"></div>
|
||||||
|
|
||||||
|
<div id="jobResult" class="hidden pt-1 flex items-center gap-3">
|
||||||
|
<a id="downloadLink" href="#" class="inline-flex items-center gap-2 rounded-xl bg-white text-black text-xs font-semibold px-5 py-2.5 hover:bg-zinc-200 transition shadow-sm">
|
||||||
|
<span>📥</span> Download EPUB
|
||||||
|
</a>
|
||||||
|
<button onclick="dismissActiveJob()" class="text-xs text-zinc-400 hover:text-zinc-200 px-3 py-2 rounded-xl border border-zinc-800 hover:bg-zinc-800/50 transition">Dismiss</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<!-- Library Section -->
|
||||||
|
<section class="rounded-2xl border border-zinc-800/80 bg-zinc-900/40 p-5 shadow-sm">
|
||||||
|
<div class="flex flex-col sm:flex-row sm:items-center justify-between gap-3 mb-4">
|
||||||
|
<div class="flex items-center gap-2">
|
||||||
|
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Library</h2>
|
||||||
|
<span id="epubCount" class="text-[11px] text-zinc-500 font-mono"></span>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-2">
|
||||||
|
<div class="relative flex-1 sm:w-60">
|
||||||
|
<input id="searchEpub" type="text" oninput="filterEpubs()" placeholder="Filter books..." class="w-full rounded-xl bg-zinc-950 border border-zinc-800 px-3 py-1.5 text-xs placeholder:text-zinc-600 focus:outline-none focus:border-zinc-600">
|
||||||
|
</div>
|
||||||
|
<button onclick="loadEpubs()" class="text-xs px-3 py-1.5 rounded-xl border border-zinc-800 hover:bg-zinc-800 hover:text-white text-zinc-400 transition shrink-0">Refresh</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div id="epubList" class="space-y-2.5"></div>
|
||||||
|
<div id="epubEmpty" class="text-xs text-zinc-500 py-8 text-center hidden">
|
||||||
|
<div class="text-2xl mb-1">📚</div>
|
||||||
|
No EPUBs found. Scrape a novel to populate your library.
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<!-- How it works Footer -->
|
||||||
|
<section class="rounded-2xl border border-zinc-800/60 bg-zinc-900/20 p-4">
|
||||||
|
<h3 class="text-xs font-semibold text-zinc-400 mb-2">Workflow Pipeline</h3>
|
||||||
|
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-2 text-[11px] text-zinc-500 leading-normal">
|
||||||
|
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
|
||||||
|
<span class="font-semibold text-zinc-400 block mb-0.5">1. Ingest & Cache</span>
|
||||||
|
Fetches novel HTML via anti-bot curl_cffi into Postgres/SQLite.
|
||||||
|
</div>
|
||||||
|
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
|
||||||
|
<span class="font-semibold text-zinc-400 block mb-0.5">2. Parse Index</span>
|
||||||
|
DeepSeek LLM extracts chronological volumes and chapter URLs.
|
||||||
|
</div>
|
||||||
|
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
|
||||||
|
<span class="font-semibold text-zinc-400 block mb-0.5">3. Sanitize Content</span>
|
||||||
|
Cleans site bloat, keeps translator notes, wraps semantic HTML.
|
||||||
|
</div>
|
||||||
|
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
|
||||||
|
<span class="font-semibold text-zinc-400 block mb-0.5">4. Build EPUB</span>
|
||||||
|
Downloads images with referers, compresses WEBP/JPG, compiles book.
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<!-- In-Page Toast Container -->
|
||||||
|
<div id="toastContainer" class="fixed bottom-4 right-4 z-50 flex flex-col gap-2 pointer-events-none"></div>
|
||||||
|
|
||||||
|
<footer class="border-t border-zinc-800/80 py-4 text-center text-[11px] text-zinc-600">
|
||||||
|
Silksoup — Self-hosted EPUB Generator • Fast, headless, and cached.
|
||||||
|
</footer>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const $ = s => document.querySelector(s);
|
||||||
|
let currentJobId = localStorage.getItem('silksoup_active_job') || null;
|
||||||
|
let evtSource = null;
|
||||||
|
let allEpubs = [];
|
||||||
|
let originalTitle = document.title;
|
||||||
|
let notificationsAllowed = Notification.permission === 'granted';
|
||||||
|
|
||||||
|
updateNotifyButton();
|
||||||
|
|
||||||
|
function updateNotifyButton(){
|
||||||
|
if (Notification.permission === 'granted') {
|
||||||
|
$('#notifyBtn').classList.add('text-emerald-400', 'border-emerald-500/30');
|
||||||
|
$('#notifyText').textContent = 'Notifications on';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function toggleNotifications(){
|
||||||
|
if (Notification.permission === 'default') {
|
||||||
|
const res = await Notification.requestPermission();
|
||||||
|
if (res === 'granted') {
|
||||||
|
showToast('Notifications enabled!', 'success');
|
||||||
|
updateNotifyButton();
|
||||||
|
}
|
||||||
|
} else if (Notification.permission === 'denied') {
|
||||||
|
showToast('Notifications are blocked in your browser settings.', 'error');
|
||||||
|
} else {
|
||||||
|
showToast('Notifications are already enabled.', 'info');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function notify(title, body){
|
||||||
|
if (Notification.permission === 'granted' && document.hidden) {
|
||||||
|
new Notification(title, { body, icon: '/static/favicon.ico' });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function showToast(message, type = 'info'){
|
||||||
|
const container = $('#toastContainer');
|
||||||
|
const toast = document.createElement('div');
|
||||||
|
const bg = type === 'error' ? 'bg-rose-950/90 border-rose-800 text-rose-200' :
|
||||||
|
type === 'success' ? 'bg-emerald-950/90 border-emerald-800 text-emerald-200' :
|
||||||
|
'bg-zinc-900/90 border-zinc-800 text-zinc-200';
|
||||||
|
toast.className = `pointer-events-auto rounded-xl border px-4 py-3 text-xs shadow-lg backdrop-blur flex items-center gap-2 transition-all transform translate-y-2 opacity-0 duration-200 ${bg}`;
|
||||||
|
toast.innerHTML = `<span>${type === 'error' ? '⚠️' : type === 'success' ? '✓' : 'ℹ️'}</span><span class="flex-1">${message}</span>`;
|
||||||
|
container.appendChild(toast);
|
||||||
|
requestAnimationFrame(() => {
|
||||||
|
toast.classList.remove('translate-y-2', 'opacity-0');
|
||||||
|
});
|
||||||
|
setTimeout(() => {
|
||||||
|
toast.classList.add('opacity-0', 'translate-y-2');
|
||||||
|
setTimeout(() => toast.remove(), 250);
|
||||||
|
}, 4000);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function pasteClipboard(){
|
||||||
|
try {
|
||||||
|
const text = await navigator.clipboard.readText();
|
||||||
|
if (text) {
|
||||||
|
$('#urlInput').value = text.trim();
|
||||||
|
showToast('Pasted URL from clipboard', 'info');
|
||||||
|
}
|
||||||
|
} catch(e) {
|
||||||
|
showToast('Clipboard access denied', 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function setBtnLoading(loading){
|
||||||
|
const btn = $('#startBtn');
|
||||||
|
const spinner = $('#btnSpinner');
|
||||||
|
const text = $('#btnText');
|
||||||
|
if (loading) {
|
||||||
|
btn.disabled = true;
|
||||||
|
spinner.classList.remove('hidden');
|
||||||
|
text.textContent = 'Submitting...';
|
||||||
|
} else {
|
||||||
|
btn.disabled = false;
|
||||||
|
spinner.classList.add('hidden');
|
||||||
|
text.textContent = 'Start Scraping → EPUB';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$('#startBtn').addEventListener('click', async () => {
|
||||||
|
const url = $('#urlInput').value.trim();
|
||||||
|
if (!url) {
|
||||||
|
showToast('Please enter a novel URL', 'error');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
setBtnLoading(true);
|
||||||
|
try {
|
||||||
|
const res = await fetch('/api/jobs', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {'Content-Type':'application/json'},
|
||||||
|
body: JSON.stringify({ url, force_refresh: $('#refreshInput').checked })
|
||||||
|
});
|
||||||
|
if (!res.ok) {
|
||||||
|
const j = await res.json();
|
||||||
|
throw new Error(j.detail || 'Failed to submit job');
|
||||||
|
}
|
||||||
|
const data = await res.json();
|
||||||
|
currentJobId = data.job_id;
|
||||||
|
localStorage.setItem('silksoup_active_job', currentJobId);
|
||||||
|
showActive();
|
||||||
|
connectSSE(currentJobId);
|
||||||
|
showToast('Job started successfully', 'success');
|
||||||
|
} catch(e) {
|
||||||
|
showToast(e.message, 'error');
|
||||||
|
} finally {
|
||||||
|
setBtnLoading(false);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
function showActive(){
|
||||||
|
$('#activeJob').classList.remove('hidden');
|
||||||
|
$('#headerBadge').classList.remove('hidden');
|
||||||
|
$('#cancelJobBtn').classList.remove('hidden');
|
||||||
|
$('#jobLog').textContent = '';
|
||||||
|
$('#jobProgress').style.width = '0%';
|
||||||
|
$('#jobCounter').textContent = '0 / 0';
|
||||||
|
$('#jobStage').textContent = 'queued';
|
||||||
|
$('#jobStatusBadge').textContent = 'running';
|
||||||
|
$('#jobResult').classList.add('hidden');
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400 animate-glow';
|
||||||
|
$('#activeJob').scrollIntoView({ behavior: 'smooth', block: 'nearest' });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function requestCancelJob(){
|
||||||
|
if (!currentJobId) return;
|
||||||
|
const btn = $('#cancelJobBtn');
|
||||||
|
btn.disabled = true;
|
||||||
|
btn.textContent = 'Stopping...';
|
||||||
|
try {
|
||||||
|
const res = await fetch(`/api/jobs/${currentJobId}/cancel`, { method: 'POST' });
|
||||||
|
if (res.ok) {
|
||||||
|
showToast('Process cancellation requested. Cached chapters remain safe.', 'info');
|
||||||
|
}
|
||||||
|
} catch(e) {
|
||||||
|
showToast('Failed to cancel job', 'error');
|
||||||
|
} finally {
|
||||||
|
btn.disabled = false;
|
||||||
|
btn.textContent = 'Stop Process';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function dismissActiveJob(){
|
||||||
|
localStorage.removeItem('silksoup_active_job');
|
||||||
|
currentJobId = null;
|
||||||
|
if (evtSource) evtSource.close();
|
||||||
|
$('#activeJob').classList.add('hidden');
|
||||||
|
$('#headerBadge').classList.add('hidden');
|
||||||
|
document.title = originalTitle;
|
||||||
|
}
|
||||||
|
|
||||||
|
function connectSSE(jobId){
|
||||||
|
if (evtSource) evtSource.close();
|
||||||
|
evtSource = new EventSource(`/api/jobs/${jobId}/events`);
|
||||||
|
|
||||||
|
evtSource.onmessage = (e) => {
|
||||||
|
if (e.data.startsWith(':')) return;
|
||||||
|
const evt = JSON.parse(e.data);
|
||||||
|
if (evt.message === '__DONE__') {
|
||||||
|
evtSource.close();
|
||||||
|
refreshJob();
|
||||||
|
loadEpubs();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const pct = evt.total ? Math.round(evt.current / evt.total * 100) : 0;
|
||||||
|
$('#jobStage').textContent = evt.stage;
|
||||||
|
$('#jobMessage').textContent = evt.message || '';
|
||||||
|
$('#jobCounter').textContent = `${evt.current} / ${evt.total}`;
|
||||||
|
$('#jobProgress').style.width = pct + '%';
|
||||||
|
|
||||||
|
if (evt.stage === 'scraping' || evt.stage === 'cleaning' || evt.stage === 'epub') {
|
||||||
|
document.title = `(${pct}%) Silksoup`;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (evt.stage === 'failed') {
|
||||||
|
$('#jobStatusBadge').textContent = 'failed';
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
document.title = `❌ Failed — Silksoup`;
|
||||||
|
showToast(`Job failed: ${evt.message}`, 'error');
|
||||||
|
}
|
||||||
|
if (evt.stage === 'cancelled') {
|
||||||
|
$('#jobStatusBadge').textContent = 'cancelled';
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
document.title = `Silksoup`;
|
||||||
|
showToast('Scraping stopped. Completed chapters are cached in database.', 'info');
|
||||||
|
}
|
||||||
|
if (evt.stage === 'completed') {
|
||||||
|
$('#jobStatusBadge').textContent = 'completed';
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
document.title = `✓ Completed — Silksoup`;
|
||||||
|
notify('Silksoup EPUB Ready!', evt.message || 'Your book has been compiled successfully.');
|
||||||
|
showToast('EPUB compiled successfully!', 'success');
|
||||||
|
}
|
||||||
|
|
||||||
|
if (evt.message && evt.message !== '__DONE__') {
|
||||||
|
const log = $('#jobLog');
|
||||||
|
log.textContent += `[${evt.stage}] ${evt.message}\n`;
|
||||||
|
if ($('#autoScrollCheck').checked) {
|
||||||
|
log.scrollTop = log.scrollHeight;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
evtSource.onerror = () => {
|
||||||
|
// Retry handled by browser EventSource
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function refreshJob(){
|
||||||
|
if (!currentJobId) return;
|
||||||
|
const res = await fetch(`/api/jobs/${currentJobId}`);
|
||||||
|
if (!res.ok) return;
|
||||||
|
const j = await res.json();
|
||||||
|
$('#jobStatusBadge').textContent = j.status;
|
||||||
|
if (j.status === 'completed') {
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
if (j.result_path) {
|
||||||
|
const name = j.result_path.split(/[\\/]/).pop();
|
||||||
|
$('#downloadLink').href = `/api/epubs/${encodeURIComponent(name)}`;
|
||||||
|
$('#downloadLink').innerHTML = `<span>📥</span> Download ${name}`;
|
||||||
|
$('#jobResult').classList.remove('hidden');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (j.status === 'cancelled') {
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
}
|
||||||
|
if (j.status === 'failed') {
|
||||||
|
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
|
||||||
|
$('#cancelJobBtn').classList.add('hidden');
|
||||||
|
$('#jobLog').textContent += `\n[ERROR] ${j.error || j.message}\n`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function clearLogs(){
|
||||||
|
$('#jobLog').textContent = '';
|
||||||
|
}
|
||||||
|
|
||||||
|
let isLogExpanded = false;
|
||||||
|
function toggleLogHeight(){
|
||||||
|
const log = $('#jobLog');
|
||||||
|
const btn = $('#expandLogBtn');
|
||||||
|
if (isLogExpanded) {
|
||||||
|
log.classList.remove('h-96');
|
||||||
|
log.classList.add('h-44');
|
||||||
|
btn.textContent = 'Expand';
|
||||||
|
isLogExpanded = false;
|
||||||
|
} else {
|
||||||
|
log.classList.remove('h-44');
|
||||||
|
log.classList.add('h-96');
|
||||||
|
btn.textContent = 'Collapse';
|
||||||
|
isLogExpanded = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadEpubs(){
|
||||||
|
try {
|
||||||
|
const res = await fetch('/api/epubs');
|
||||||
|
const data = await res.json();
|
||||||
|
allEpubs = data.epubs || [];
|
||||||
|
$('#epubCount').textContent = allEpubs.length ? `(${allEpubs.length})` : '';
|
||||||
|
renderEpubs(allEpubs);
|
||||||
|
} catch(e) {
|
||||||
|
showToast('Failed to load library', 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function filterEpubs(){
|
||||||
|
const query = $('#searchEpub').value.toLowerCase().trim();
|
||||||
|
if (!query) {
|
||||||
|
renderEpubs(allEpubs);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const filtered = allEpubs.filter(f => f.name.toLowerCase().includes(query));
|
||||||
|
renderEpubs(filtered);
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderEpubs(list){
|
||||||
|
const container = $('#epubList');
|
||||||
|
container.innerHTML = '';
|
||||||
|
if (!list.length) {
|
||||||
|
$('#epubEmpty').classList.remove('hidden');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$('#epubEmpty').classList.add('hidden');
|
||||||
|
|
||||||
|
for (const f of list) {
|
||||||
|
const row = document.createElement('div');
|
||||||
|
row.className = 'group flex items-center gap-3.5 rounded-xl border border-zinc-800/80 bg-zinc-950/80 px-4 py-3 hover:border-zinc-700 transition';
|
||||||
|
|
||||||
|
// Clean filename for nice display
|
||||||
|
const cleanTitle = f.name.replace(/\.epub$/i, '').replace(/[-_]/g, ' ');
|
||||||
|
|
||||||
|
row.innerHTML = `
|
||||||
|
<div class="w-8 h-8 rounded-lg bg-zinc-900 border border-zinc-800 flex items-center justify-center text-xs shrink-0 text-zinc-400 group-hover:text-white transition">📖</div>
|
||||||
|
<div class="flex-1 min-w-0">
|
||||||
|
<div class="truncate text-zinc-200 text-xs font-medium capitalize" title="${cleanTitle}">${cleanTitle}</div>
|
||||||
|
<div class="text-[11px] text-zinc-400 font-mono mt-0.5">${f.size_human} • ${new Date(f.mtime*1000).toLocaleDateString()} ${new Date(f.mtime*1000).toLocaleTimeString([], {hour: '2-digit', minute:'2-digit'})}</div>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-2 shrink-0">
|
||||||
|
<a href="/api/epubs/${encodeURIComponent(f.name)}" download class="text-xs px-3.5 py-1.5 rounded-xl bg-white text-black font-semibold hover:bg-zinc-200 transition">Download</a>
|
||||||
|
<div class="relative inline-block" id="delWrap_${CSS.escape(f.name)}">
|
||||||
|
<button onclick="confirmDelete('${encodeURIComponent(f.name)}')" class="text-xs px-2.5 py-1.5 rounded-xl border border-zinc-800 hover:border-rose-900/60 hover:text-rose-400 text-zinc-400 transition">Delete</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
container.appendChild(row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function confirmDelete(encodedName){
|
||||||
|
const name = decodeURIComponent(encodedName);
|
||||||
|
const wrap = document.getElementById(`delWrap_${CSS.escape(name)}`);
|
||||||
|
if (!wrap) return;
|
||||||
|
wrap.innerHTML = `
|
||||||
|
<div class="flex items-center gap-1.5 bg-rose-950/60 border border-rose-800/80 rounded-xl p-0.5">
|
||||||
|
<button onclick="executeDelete('${encodeURIComponent(name)}')" class="text-[11px] px-2 py-1 bg-rose-600 text-white rounded-lg font-medium hover:bg-rose-500">Confirm</button>
|
||||||
|
<button onclick="loadEpubs()" class="text-[11px] px-2 py-1 text-zinc-400 hover:text-white">Cancel</button>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function executeDelete(encodedName){
|
||||||
|
try {
|
||||||
|
const res = await fetch(`/api/epubs/${encodedName}`, { method: 'DELETE' });
|
||||||
|
if (!res.ok) throw new Error('Delete failed');
|
||||||
|
showToast('EPUB removed from library', 'info');
|
||||||
|
loadEpubs();
|
||||||
|
} catch(e) {
|
||||||
|
showToast('Failed to delete file', 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Auto-restore active job on load if any
|
||||||
|
async function checkStoredJob(){
|
||||||
|
if (!currentJobId) return;
|
||||||
|
try {
|
||||||
|
const res = await fetch(`/api/jobs/${currentJobId}`);
|
||||||
|
if (res.ok) {
|
||||||
|
const j = await res.json();
|
||||||
|
showActive();
|
||||||
|
$('#jobStage').textContent = j.stage;
|
||||||
|
$('#jobMessage').textContent = j.message || '';
|
||||||
|
$('#jobCounter').textContent = `${j.current} / ${j.total}`;
|
||||||
|
const pct = j.total ? Math.round(j.current / j.total * 100) : 0;
|
||||||
|
$('#jobProgress').style.width = pct + '%';
|
||||||
|
if (j.logs) {
|
||||||
|
$('#jobLog').textContent = j.logs.join('\n') + '\n';
|
||||||
|
}
|
||||||
|
if (j.status === 'running' || j.status === 'queued') {
|
||||||
|
connectSSE(currentJobId);
|
||||||
|
} else {
|
||||||
|
refreshJob();
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
localStorage.removeItem('silksoup_active_job');
|
||||||
|
}
|
||||||
|
} catch(e) {
|
||||||
|
localStorage.removeItem('silksoup_active_job');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
checkStoredJob();
|
||||||
|
loadEpubs();
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
Reference in New Issue
Block a user