Compare commits

...
10 Commits
Author SHA1 Message Date
Sylphian bd3d978216 feat: add a stop process button 2026-09-06 17:11:35 +07:00
Sylphian 2581e01e3b feat: ui/ux improvements 2026-09-06 17:00:35 +07:00
Sylphian 648c15a7ef Version 2.0 2026-09-06 16:31:03 +07:00
Sylphian 361ff407f0 feat: add referer headers for image downloads and webp compression 2026-09-06 15:50:46 +07:00
Sylphian 948c826acf Add a hidden ajax chapter list detection and a no-cache flag 2026-08-25 22:22:46 +07:00
Sylphian 0314ebb931 Pass light novel link through CLI 2026-08-25 19:31:46 +07:00
Sylphian 2ebd54cc6f Add output directory to gitignore 2026-08-23 22:01:21 +07:00
Sylphian cf9a6cfd0d Version 1.0 2026-08-23 19:36:25 +07:00
Sylphian 9f15215c26 Pre Change 2026-08-23 18:50:58 +07:00
Sylphian 00d751f99f Pre fixing 2026-08-23 18:47:07 +07:00
18 changed files with 1691 additions and 144 deletions
+5
View File
@@ -0,0 +1,5 @@
LLM_API_KEY=your_api_key_here
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=ds/deepseek-v4-flash
DATABASE_URL=postgresql://user:password@localhost:5432/silksoup
OUTPUT_DIR=output
+1
View File
@@ -4,3 +4,4 @@ __pycache__/
*.pyc *.pyc
sqlite.db sqlite.db
.env .env
output/
+18
View File
@@ -0,0 +1,18 @@
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential curl libxml2-dev libxslt1-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN mkdir -p output
EXPOSE 8000
CMD ["uvicorn", "web.app:app", "--host", "0.0.0.0", "--port", "8000"]
+122
View File
@@ -0,0 +1,122 @@
# Silksoup
> Turn online web novels and light novels into clean, beautifully formatted EPUB e-books with a single click or command.
Silksoup scrapes web novel index pages, uses an OpenAI-compatible LLM to detect volumes and chapters, cleans out site ads and junk, downloads illustrations, and packages everything into an offline `.epub` file ready for your e-reader, phone, or tablet.
---
## Key Features
- **Dual Modes:** Use the visual **Web Dashboard** or the terminal **CLI tool**—both share the same engine.
- **Smart Chapter Extraction:** Uses AI to understand complex table of contents, volumes, and reading order.
- **Clean Reading Experience:** Strips navigation links, donation banners, and website watermarks while preserving important translator notes, formatting, and character dialogue.
- **Image & Illustration Support:** Downloads cover art and embedded chapter illustrations (supports WEBP, JPEG, PNG, GIF) with anti-hotlinking headers.
- **Smart Caching (PostgreSQL & SQLite):** Stores downloaded chapters so re-runs and updates don't re-download what you already have.
- **Self-Hostable & Docker Ready:** Run it locally or host it on a VPS behind WireGuard / VPN.
---
## Quick Start (Web App)
### Option A: Using Docker (Recommended for Servers / VPS)
1. **Clone the repository:**
```bash
git clone https://github.com/your-username/silksoup.git
cd silksoup
```
2. **Set up your environment variables:**
```bash
cp .env.example .env
```
Open `.env` and configure your OpenAI-compatible LLM:
```env
LLM_API_KEY=your_api_key_here
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=ds/deepseek-v4-flash
# Optional: Connect to your PostgreSQL database (leave blank to use local SQLite)
DATABASE_URL=postgresql://user:password@localhost:5432/silksoup
```
3. **Start the service:**
```bash
docker compose up -d --build
```
4. **Open your browser:**
Go to `http://localhost:8000` (or `http://YOUR_SERVER_IP:8000`).
---
### Option B: Running Locally with Python
1. **Install dependencies:**
```bash
pip install -r requirements.txt
# or with uv:
uv pip install -r requirements.txt
```
2. **Configure `.env`** (see step 2 above).
3. **Start the web server:**
```bash
python server.py
```
Open `http://localhost:8000` in your browser.
---
## Web Dashboard Guide
1. **Enter Novel URL:** Paste the main index or table-of-contents link of the novel into the input box.
2. **Start Scraping:** Click **"Start Scraping -> EPUB"**.
3. **Live Progress:** Watch real-time progress as chapters are fetched, cleaned, and compiled into the EPUB.
4. **Download & Manage:** Once complete, click **"Download EPUB"** or manage your compiled books directly from the built-in **Library**.
---
## CLI Usage
If you prefer working from the terminal, run `main.py` directly:
### CLI Syntax:
```bash
python main.py <url> [-o OUTPUT] [--refresh]
```
### Flags & Options:
| Option / Flag | Type | Description |
|---|---|---|
| `url` | *Positional (Required)* | The main index / table-of-contents URL of the novel. |
| `-o`, `--output` | *Optional* | Output directory for the compiled EPUB file (default: `./output`). |
| `--refresh` | *Optional (Flag)* | Force refresh cached pages and metadata instead of using SQLite/PostgreSQL cache. |
| `-h`, `--help` | *Optional (Flag)* | Show command-line help message and exit. |
---
## Where Are My EPUB Files?
All compiled `.epub` files are saved directly inside the `output/` directory in the project folder:
- **Host Path:** `./output/<novel-title>.epub`
- **Docker Path:** `/app/output/<novel-title>.epub`
You can move these files to your e-reader (Kindle, Kobo, Boox) or integrate the folder with media servers like **Calibre-Web**, **Kavita**, or **Audiobookshelf**.
---
## Configuration Reference
All settings can be configured inside your `.env` file:
| Variable | Description | Default |
|---|---|---|
| `LLM_API_KEY` | **Required.** Your API key for the OpenAI-compatible LLM. | *None* |
| `LLM_BASE_URL` | **Required.** Base URL of any OpenAI-compatible API endpoint. | *None* |
| `LLM_MODEL` | **Required.** Model identifier used for extraction & cleaning. | *None* |
| `DATABASE_URL` | PostgreSQL connection string. If omitted, uses local `sqlite.db`. | *None (SQLite)* |
| `OUTPUT_DIR` | Directory where compiled EPUBs are stored. | `output` |
+17
View File
@@ -0,0 +1,17 @@
services:
app:
build: .
container_name: silksoup
ports:
- "127.0.0.1:8000:8000"
env_file:
- .env
environment:
- DATABASE_URL=${DATABASE_URL:-}
- LLM_API_KEY=${LLM_API_KEY}
- LLM_BASE_URL=${LLM_BASE_URL}
- LLM_MODEL=${LLM_MODEL}
- OUTPUT_DIR=/app/output
volumes:
- ./output:/app/output
restart: unless-stopped
+36 -5
View File
@@ -1,18 +1,49 @@
import argparse
from pathlib import Path
from src.database import init_db from src.database import init_db
from src.metadata import extract_metadata_json from src.metadata import extract_metadata_json
from src.parse import cache_hit from src.parse import cache_hit
from src.clean_html import clean_novel_html from src.clean_html import clean_novel_html
from src.chapter import clean_chapters from src.chapter import clean_chapters
from src.epub_creator import create_epub
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Scrape light novel and compile EPUB."
)
parser.add_argument(
"url",
type=str,
help="URL of the novel table of contents / index page",
)
parser.add_argument(
"-o",
"--output",
type=str,
default=str(Path(__file__).resolve().parent / "output"),
help="Output directory for the compiled EPUB (default: ./output)",
)
parser.add_argument(
"--refresh",
action="store_true",
help="Force refresh cached pages/metadata instead of using SQLite cache",
)
return parser.parse_args()
def main() -> None: def main() -> None:
args = parse_args()
init_db() init_db()
page_url = "https://hikkicave.site/story/i-became-friends-with-the-second-cutest-girl-in-my-class-wn/" html_content = clean_novel_html(
cache_hit(args.url, force_refresh=args.refresh)
html_content = clean_novel_html(cache_hit(page_url)) )
metadata = extract_metadata_json(html_content, page_url) metadata = extract_metadata_json(
clean_chapters(metadata) html_content, args.url, force_refresh=args.refresh
)
clean_chapters(metadata, force_refresh=args.refresh)
create_epub(metadata, output_dir=args.output)
if __name__ == "__main__": if __name__ == "__main__":
+14
View File
@@ -0,0 +1,14 @@
beautifulsoup4==4.15.0
lxml==6.1.2
curl-cffi==0.16.1
ebooklib==0.20
Pillow==12.3.0
tqdm==4.70.0
openai==3.3.1
python-dotenv==1.2.3
fastapi==0.115.0
uvicorn[standard]==0.30.6
jinja2==3.1.4
psycopg[binary]==3.2.3
python-multipart==0.0.9
httpx==0.28.1
+4
View File
@@ -0,0 +1,4 @@
import uvicorn
if __name__ == "__main__":
uvicorn.run("web.app:app", host="0.0.0.0", port=8000, reload=False)
+62 -52
View File
@@ -1,15 +1,9 @@
import sqlite3 from tqdm import tqdm
from google import genai from .database import db_execute, db_fetch_one
from google.genai import types
from dotenv import load_dotenv
from .database import DB_PATH
from .parse import robust_fetch from .parse import robust_fetch
from .clean_html import clean_novel_html from .clean_html import clean_novel_html
from .prompt import HTML_CHAPTER_SANITIZER from .prompt import HTML_CHAPTER_SANITIZER
from .llm import generate_text
load_dotenv()
client = genai.Client()
def iter_chapters(metadata: dict): def iter_chapters(metadata: dict):
@@ -24,80 +18,96 @@ def iter_chapters(metadata: dict):
def sanitize_chapter_html(html: str) -> str: def sanitize_chapter_html(html: str) -> str:
response = client.models.generate_content( return generate_text(HTML_CHAPTER_SANITIZER, html)
model="gemini-3.6-flash",
contents=html,
config=types.GenerateContentConfig(
system_instruction=HTML_CHAPTER_SANITIZER,
temperature=0.0,
response_mime_type="text/plain",
tools=None,
),
)
return response.text.strip()
def cache_chapter_hit(url: str) -> str: def cache_chapter_hit(url: str, force_refresh: bool = False) -> str:
with sqlite3.connect(DB_PATH) as conn: if not force_refresh:
cursor = conn.cursor() row = db_fetch_one("SELECT html_text FROM chapter_page WHERE url = ?", (url,))
cursor.execute("SELECT html_text FROM chapter_page WHERE url = ?", (url,))
row = cursor.fetchone()
if row is not None and row[0] is not None: if row is not None and row[0] is not None:
print(f"Chapter cache hit: {url}") tqdm.write(f"Chapter cache hit: {url}")
return str(row[0]) return str(row[0])
status_code, html = robust_fetch(url) status_code, html = robust_fetch(url)
if status_code != 200: if status_code != 200:
raise RuntimeError(f"Request to {url} failed with status code {status_code}") raise RuntimeError(f"Request to {url} failed with status code {status_code}")
with sqlite3.connect(DB_PATH) as conn: db_execute(
conn.execute( "INSERT INTO chapter_page (url, status_code, html_text) VALUES (?, ?, ?) ON CONFLICT(url) DO UPDATE SET status_code=excluded.status_code, html_text=excluded.html_text, fetched_at=CURRENT_TIMESTAMP",
"INSERT OR REPLACE INTO chapter_page (url, status_code, html_text) VALUES (?, ?, ?)",
(url, status_code, html), (url, status_code, html),
) )
conn.commit()
return html return html
def cache_clean_hit(url: str) -> str: def cache_clean_hit(url: str, force_refresh: bool = False) -> str:
with sqlite3.connect(DB_PATH) as conn: if not force_refresh:
cursor = conn.cursor() row = db_fetch_one("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
cursor.execute("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
row = cursor.fetchone()
if row is not None and row[0] is not None: if row is not None and row[0] is not None:
print(f"Clean HTML cache hit: {url}") tqdm.write(f"Clean HTML cache hit: {url}")
return str(row[0]) return str(row[0])
raw_html = cache_chapter_hit(url) raw_html = cache_chapter_hit(url, force_refresh=force_refresh)
pre_cleaned = clean_novel_html(raw_html, base_url=url) pre_cleaned = clean_novel_html(raw_html, base_url=url)
sanitized = sanitize_chapter_html(pre_cleaned) sanitized = sanitize_chapter_html(pre_cleaned)
with sqlite3.connect(DB_PATH) as conn: db_execute(
conn.execute(
"UPDATE chapter_page SET clean_html_text = ? WHERE url = ?", "UPDATE chapter_page SET clean_html_text = ? WHERE url = ?",
(sanitized, url), (sanitized, url),
) )
conn.commit()
return sanitized return sanitized
def scrape_chapters(metadata: dict) -> None: def scrape_chapters(
for chapter in iter_chapters(metadata): metadata: dict,
print(f"[{chapter['volume']}] {chapter['name']} -> {chapter['url']}") force_refresh: bool = False,
progress_callback=None,
cancel_event=None,
) -> None:
chapters = list(iter_chapters(metadata))
total = len(chapters)
if progress_callback is None:
iterator = tqdm(chapters, desc="Scraping chapters", unit="ch")
else:
iterator = chapters
for idx, chapter in enumerate(iterator, start=1):
if cancel_event is not None and cancel_event.is_set():
raise InterruptedError("Scraping was cancelled by user")
try: try:
cache_chapter_hit(chapter["url"]) cache_chapter_hit(chapter["url"], force_refresh=force_refresh)
if progress_callback:
progress_callback(stage="scraping", current=idx, total=total, message=chapter.get("name") or chapter.get("url") or "")
except InterruptedError:
raise
except Exception as e: except Exception as e:
print(f"[SKIP] {chapter['name']}: {e}") tqdm.write(f"[SKIP] {chapter['name']}: {e}")
continue if progress_callback:
progress_callback(stage="scraping", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")
def clean_chapters(metadata: dict) -> None: def clean_chapters(
for chapter in iter_chapters(metadata): metadata: dict,
print(f"[{chapter['volume']}] Cleaning: {chapter['name']}") force_refresh: bool = False,
progress_callback=None,
cancel_event=None,
) -> None:
chapters = list(iter_chapters(metadata))
total = len(chapters)
if progress_callback is None:
iterator = tqdm(chapters, desc="Cleaning chapters", unit="ch")
else:
iterator = chapters
for idx, chapter in enumerate(iterator, start=1):
if cancel_event is not None and cancel_event.is_set():
raise InterruptedError("Cleaning was cancelled by user")
try: try:
cache_clean_hit(chapter["url"]) cache_clean_hit(chapter["url"], force_refresh=force_refresh)
if progress_callback:
progress_callback(stage="cleaning", current=idx, total=total, message=chapter.get("name") or chapter.get("url") or "")
except InterruptedError:
raise
except Exception as e: except Exception as e:
print(f"[SKIP] {chapter['name']}: {e}") tqdm.write(f"[SKIP] {chapter['name']}: {e}")
continue if progress_callback:
progress_callback(stage="cleaning", current=idx, total=total, message=f"[SKIP] {chapter.get('name')}: {e}")
+85 -2
View File
@@ -1,14 +1,98 @@
import os
import sqlite3 import sqlite3
from contextlib import contextmanager
from pathlib import Path from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent.parent SCRIPT_DIR = Path(__file__).resolve().parent.parent
DB_PATH = str(SCRIPT_DIR / "sqlite.db") DB_PATH = str(SCRIPT_DIR / "sqlite.db")
DATABASE_URL = os.environ.get("DATABASE_URL", "").strip()
USE_PG = DATABASE_URL.startswith("postgres://") or DATABASE_URL.startswith("postgresql://")
if USE_PG and DATABASE_URL.startswith("postgres://"):
DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
def _adapt_query(query: str) -> str:
if USE_PG:
return query.replace("?", "%s")
return query
@contextmanager
def get_conn():
if USE_PG:
import psycopg
conn = psycopg.connect(DATABASE_URL)
try:
yield conn
conn.commit()
finally:
conn.close()
else:
conn = sqlite3.connect(DB_PATH)
try:
yield conn
conn.commit()
finally:
conn.close()
def db_fetch_one(query: str, params: tuple = ()):
q = _adapt_query(query)
with get_conn() as conn:
cur = conn.cursor()
cur.execute(q, params)
return cur.fetchone()
def db_fetch_all(query: str, params: tuple = ()):
q = _adapt_query(query)
with get_conn() as conn:
cur = conn.cursor()
cur.execute(q, params)
return cur.fetchall()
def db_execute(query: str, params: tuple = ()):
q = _adapt_query(query)
with get_conn() as conn:
cur = conn.cursor()
cur.execute(q, params)
def init_db() -> None: def init_db() -> None:
if USE_PG:
import psycopg
conn = psycopg.connect(DATABASE_URL)
try:
with conn.cursor() as cur:
cur.execute("""
CREATE TABLE IF NOT EXISTS main_page (
url TEXT PRIMARY KEY,
status_code INTEGER,
html_text TEXT,
json_text TEXT,
fetched_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
)
""")
cur.execute("""
CREATE TABLE IF NOT EXISTS chapter_page (
url TEXT PRIMARY KEY,
status_code INTEGER,
html_text TEXT,
clean_html_text TEXT,
fetched_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP
)
""")
conn.commit()
finally:
conn.close()
else:
with sqlite3.connect(DB_PATH) as conn: with sqlite3.connect(DB_PATH) as conn:
conn.execute("PRAGMA journal_mode=WAL;") conn.execute("PRAGMA journal_mode=WAL;")
conn.execute("PRAGMA synchronous=NORMAL;") conn.execute("PRAGMA synchronous=NORMAL;")
conn.execute(""" conn.execute("""
CREATE TABLE IF NOT EXISTS main_page ( CREATE TABLE IF NOT EXISTS main_page (
url TEXT PRIMARY KEY, url TEXT PRIMARY KEY,
@@ -18,7 +102,6 @@ def init_db() -> None:
fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) )
""") """)
conn.execute(""" conn.execute("""
CREATE TABLE IF NOT EXISTS chapter_page ( CREATE TABLE IF NOT EXISTS chapter_page (
url TEXT PRIMARY KEY, url TEXT PRIMARY KEY,
+251
View File
@@ -0,0 +1,251 @@
import os
import re
from io import BytesIO
from bs4 import BeautifulSoup
from ebooklib import epub
from PIL import Image
from tqdm import tqdm
from .database import db_fetch_one
from .parse import robust_fetch_bytes
_STYLE = """
@namespace epub "http://www.idpf.org/2007/ops";
body {
font-family: sans-serif;
font-size: 100%;
text-align: left;
padding-left: 5%;
padding-right: 5%;
line-height: 1.5;
}
p {
margin-top: 0;
margin-bottom: 1em;
text-indent: 1.5em;
}
h1 {
text-align: center;
text-transform: uppercase;
margin-bottom: 2em;
}
img {
max-width: 100%;
height: auto;
display: block;
margin: 1em auto;
}
hr {
border: 0;
border-top: 1px solid #ccc;
margin: 2em 0;
}
aside.tl-note {
font-size: 90%;
margin: 1em 2em;
padding: 0.5em 1em;
border-left: 3px solid #ccc;
color: #555;
}
"""
def _slugify(text: str) -> str:
slug = re.sub(r"[^\w\s-]", "", text.lower())
slug = re.sub(r"[-\s]+", "-", slug).strip("-_")
return slug or "novel"
def _get_clean_html(url: str) -> str:
row = db_fetch_one("SELECT clean_html_text FROM chapter_page WHERE url = ?", (url,))
if row is None or row[0] is None:
raise RuntimeError(
f"No cleaned HTML cached for {url}. Run clean_chapters first."
)
return str(row[0])
def _download(url: str, referer: str | None = None) -> bytes:
status_code, content = robust_fetch_bytes(url, referer=referer)
if status_code != 200:
raise RuntimeError(f"Request to {url} failed with status code {status_code}")
return content
def _compress_image(data: bytes) -> tuple[bytes, str]:
with Image.open(BytesIO(data)) as image:
image.load()
fmt = (image.format or "").upper()
buffer = BytesIO()
if fmt in ("JPEG", "JPG"):
image.save(buffer, format="JPEG", quality="keep")
return buffer.getvalue(), "jpg"
if fmt == "WEBP":
image.save(buffer, format="WEBP", quality=85)
return buffer.getvalue(), "webp"
if fmt == "PNG":
image.save(buffer, format="PNG", optimize=True)
return buffer.getvalue(), "png"
if fmt == "GIF":
image.save(buffer, format="GIF", optimize=True)
return buffer.getvalue(), "gif"
image.save(buffer, format="PNG", optimize=True)
return buffer.getvalue(), "png"
def _embed_images(
book: epub.EpubBook, html: str, image_cache: dict, referer: str | None = None
) -> str:
soup = BeautifulSoup(html, "lxml")
for img_tag in soup.find_all("img"):
src = img_tag.get("src")
if not src:
continue
if src in image_cache:
img_tag["src"] = image_cache[src]
continue
try:
raw = _download(src, referer=referer)
content, ext = _compress_image(raw)
except Exception as e:
tqdm.write(f"Error downloading/compressing image {src}: {e}")
continue
image_id = f"img_{len(image_cache) + 1}"
file_name = f"images/{image_id}.{ext}"
epub_img = epub.EpubImage()
epub_img.uid = image_id
epub_img.file_name = file_name
epub_img.content = content
book.add_item(epub_img)
img_tag["src"] = file_name
image_cache[src] = file_name
body = soup.body if soup.body is not None else soup
return body.decode_contents()
def create_epub(
metadata: dict,
output_dir: str = "output",
progress_callback=None,
cancel_event=None,
) -> str:
series = metadata.get("metadata", {})
title = series.get("title") or "Untitled"
author = series.get("author") or "Unknown"
language = series.get("language") or "en"
cover_url = series.get("cover_url")
book = epub.EpubBook()
book.set_identifier(_slugify(title))
book.set_title(title)
book.set_language(language)
book.add_author(author)
if cover_url:
try:
cover_raw = _download(cover_url, referer=metadata.get("novel_url"))
cover_bytes, cover_ext = _compress_image(cover_raw)
book.set_cover(f"cover.{cover_ext}", cover_bytes)
except Exception as e:
print(f"Cover download failed ({cover_url}): {e}. Proceeding without it.")
nav_css = epub.EpubItem(
uid="style_nav",
file_name="style/nav.css",
media_type="text/css",
content=_STYLE,
)
book.add_item(nav_css)
image_cache = {}
toc_structure = []
spine_structure = ["nav"]
chapter_counter = 1
volumes = metadata.get("volumes", [])
total_chapters = sum(len(v.get("chapters", [])) for v in volumes)
def _process_volume(volume, pbar=None):
nonlocal chapter_counter
volume_name = volume.get("name", "Volume")
sections = []
for chapter in volume.get("chapters", []):
if cancel_event is not None and cancel_event.is_set():
raise InterruptedError("EPUB compilation was cancelled by user")
chapter_name = chapter.get("name") or "Untitled"
url = chapter.get("url")
if pbar is not None:
pbar.set_postfix_str(chapter_name[:30])
pbar.update(1)
if progress_callback is not None:
progress_callback(stage="epub", current=chapter_counter, total=total_chapters, message=chapter_name)
if not url:
tqdm.write(f"[SKIP] No URL for chapter: {chapter_name}")
chapter_counter += 1
continue
try:
clean_html = _get_clean_html(url)
except RuntimeError as e:
tqdm.write(f"[SKIP] {chapter_name}: {e}")
chapter_counter += 1
continue
clean_html = _embed_images(book, clean_html, image_cache, referer=url)
file_name = f"chapter_{chapter_counter}.xhtml"
epub_chapter = epub.EpubHtml(
title=chapter_name,
file_name=file_name,
lang=language,
)
epub_chapter.add_item(nav_css)
epub_chapter.content = f"<h1>{chapter_name}</h1>\n{clean_html}"
book.add_item(epub_chapter)
sections.append(epub_chapter)
spine_structure.append(epub_chapter)
chapter_counter += 1
if sections:
toc_structure.append((epub.Section(volume_name), tuple(sections)))
if progress_callback is None:
with tqdm(total=total_chapters, desc="Building EPUB", unit="ch") as pbar:
for volume in volumes:
_process_volume(volume, pbar=pbar)
else:
for volume in volumes:
_process_volume(volume, pbar=None)
if not spine_structure[1:]:
raise RuntimeError("No chapters were added to the EPUB.")
book.toc = toc_structure
book.add_item(epub.EpubNav())
book.add_item(epub.EpubNcx())
book.spine = spine_structure
os.makedirs(output_dir, exist_ok=True)
output_path = os.path.join(output_dir, f"{_slugify(title)}.epub")
epub.write_epub(output_path, book, {})
print(f"Successfully generated: {output_path}")
return output_path
+51
View File
@@ -0,0 +1,51 @@
import os
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
MODEL = os.environ.get("LLM_MODEL")
if not MODEL:
raise RuntimeError("LLM_MODEL is not set. Add it to your .env file.")
BASE_URL = os.environ.get("LLM_BASE_URL")
if not BASE_URL:
raise RuntimeError("LLM_BASE_URL is not set. Add it to your .env file.")
MAX_TOKENS = int(os.environ.get("LLM_MAX_TOKENS", "65536"))
API_KEY = os.environ.get("LLM_API_KEY")
if not API_KEY:
raise RuntimeError("LLM_API_KEY is not set. Add it to your .env file.")
client = OpenAI(
api_key=API_KEY,
base_url=BASE_URL,
)
def _request(system_prompt: str, user_content: str, response_format=None) -> str:
kwargs = {
"model": MODEL,
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_content},
],
"temperature": 0.0,
"max_tokens": MAX_TOKENS,
"extra_body": {"thinking": {"type": "disabled"}},
}
if response_format is not None:
kwargs["response_format"] = response_format
response = client.chat.completions.create(**kwargs)
return (response.choices[0].message.content or "").strip()
def generate_json(system_prompt: str, user_content: str) -> str:
return _request(system_prompt, user_content, response_format={"type": "json_object"})
def generate_text(system_prompt: str, user_content: str) -> str:
return _request(system_prompt, user_content)
+16 -34
View File
@@ -1,57 +1,39 @@
import json import json
import os from .database import db_execute, db_fetch_one
import re
import sqlite3
from google import genai
from google.genai import types
from dotenv import load_dotenv
from .database import DB_PATH
from .parse import cache_hit from .parse import cache_hit
from .clean_html import clean_novel_html from .clean_html import clean_novel_html
from .prompt import INDEX_EXTRACTION_SYSTEM_PROMPT from .prompt import INDEX_EXTRACTION_SYSTEM_PROMPT
from .llm import generate_json
load_dotenv() def extract_metadata_json(
client = genai.Client() html_content: str, url: str, force_refresh: bool = False
) -> dict:
if not force_refresh:
def extract_metadata_json(html_content: str, url: str) -> dict: row = db_fetch_one("SELECT json_text FROM main_page WHERE url = ?", (url,))
with sqlite3.connect(DB_PATH) as conn:
cursor = conn.cursor()
cursor.execute("SELECT json_text FROM main_page WHERE url = ?", (url,))
row = cursor.fetchone()
if row is not None and row[0] is not None: if row is not None and row[0] is not None:
print("Metadata JSON Cache hit!")
return json.loads(row[0]) return json.loads(row[0])
response = client.models.generate_content( raw_output = generate_json(INDEX_EXTRACTION_SYSTEM_PROMPT, html_content)
model="gemini-3.6-flash",
contents=html_content,
config=types.GenerateContentConfig(
system_instruction=INDEX_EXTRACTION_SYSTEM_PROMPT,
temperature=0.0,
response_mime_type="application/json",
tools=None
),
)
raw_output = response.text.strip()
with sqlite3.connect(DB_PATH) as conn: db_execute("""
conn.execute("""
INSERT INTO main_page (url, json_text) INSERT INTO main_page (url, json_text)
VALUES (?, ?) VALUES (?, ?)
ON CONFLICT(url) DO UPDATE SET json_text = excluded.json_text ON CONFLICT(url) DO UPDATE SET json_text = excluded.json_text
""", (url, raw_output)) """, (url, raw_output))
conn.commit()
return json.loads(raw_output) return json.loads(raw_output)
def write_json(page_url: str, output_json_path: str) -> None: def write_json(
html_content = clean_novel_html(cache_hit(page_url)) page_url: str, output_json_path: str, force_refresh: bool = False
) -> None:
html_content = clean_novel_html(cache_hit(page_url, force_refresh=force_refresh))
print("Extracting metadata via LLM...") print("Extracting metadata via LLM...")
extracted_data = extract_metadata_json(html_content, page_url) extracted_data = extract_metadata_json(
html_content, page_url, force_refresh=force_refresh
)
with open(output_json_path, "w", encoding="utf-8") as f: with open(output_json_path, "w", encoding="utf-8") as f:
json.dump(extracted_data, f, indent=2, ensure_ascii=False) json.dump(extracted_data, f, indent=2, ensure_ascii=False)
print(f"\n[SUCCESS] Extracted JSON saved to: {output_json_path}\n") print(f"\n[SUCCESS] Extracted JSON saved to: {output_json_path}\n")
+94 -21
View File
@@ -1,17 +1,22 @@
import random import random
import re
import time import time
import sqlite3 from urllib.parse import urljoin
from curl_cffi import requests from curl_cffi import requests
from curl_cffi.requests.errors import RequestsError from curl_cffi.requests.errors import RequestsError
from .database import DB_PATH from .database import db_execute, db_fetch_one
from pathlib import Path
session = requests.Session(impersonate="chrome120") session = requests.Session(impersonate="chrome120")
def robust_fetch(
url: str, min_delay: float = 2.0, max_delay: float = 5.0, retries: int = 3 def _fetch_response(
) -> str: url: str,
min_delay: float,
max_delay: float,
retries: int,
headers: dict | None = None,
):
sleep_time = random.uniform(min_delay, max_delay) sleep_time = random.uniform(min_delay, max_delay)
if random.random() < 0.08: if random.random() < 0.08:
sleep_time += random.uniform(4.0, 8.0) sleep_time += random.uniform(4.0, 8.0)
@@ -19,7 +24,7 @@ def robust_fetch(
for attempt in range(retries): for attempt in range(retries):
try: try:
response = session.get(url, timeout=15) response = session.get(url, headers=headers, timeout=15)
if response.status_code in (429, 503): if response.status_code in (429, 503):
retry_after = response.headers.get("Retry-After") retry_after = response.headers.get("Retry-After")
@@ -36,7 +41,7 @@ def robust_fetch(
"Cloudflare Managed Challenge encountered. Headless browser required." "Cloudflare Managed Challenge encountered. Headless browser required."
) )
return response.status_code, str(response.text) return response
except (RequestsError, TimeoutError) as e: except (RequestsError, TimeoutError) as e:
if attempt == retries - 1: if attempt == retries - 1:
@@ -48,25 +53,93 @@ def robust_fetch(
) )
def cache_hit(url: str) -> str: def robust_fetch(
with sqlite3.connect(DB_PATH) as conn: url: str,
cursor = conn.cursor() min_delay: float = 2.0,
cursor.execute("SELECT html_text FROM main_page WHERE url = ?", (url,)) max_delay: float = 5.0,
row = cursor.fetchone() retries: int = 3,
if row is not None: headers: dict | None = None,
print("Request cache hit!") ) -> tuple[int, str]:
response = _fetch_response(url, min_delay, max_delay, retries, headers=headers)
return response.status_code, str(response.text)
def robust_fetch_bytes(
url: str,
min_delay: float = 2.0,
max_delay: float = 5.0,
retries: int = 3,
headers: dict | None = None,
referer: str | None = None,
) -> tuple[int, bytes]:
req_headers = {
"Accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"Sec-Fetch-Dest": "image",
"Sec-Fetch-Mode": "no-cors",
"Sec-Fetch-Site": "cross-site",
}
if referer:
req_headers["Referer"] = referer
if headers:
req_headers.update(headers)
response = _fetch_response(
url, min_delay, max_delay, retries, headers=req_headers
)
return response.status_code, response.content
# Exclusive to ZTL
def _fetch_wp_manga_chapters(base_url: str, html: str) -> str:
manga_id_match = (
re.search(r'["\']manga_id["\']\s*:\s*["\']?(\d+)', html)
or re.search(r'["\']wc_post_id["\']\s*:\s*["\']?(\d+)', html)
or re.search(r'postid-(\d+)', html)
)
if not manga_id_match:
return ""
manga_id = manga_id_match.group(1)
ajax_url_match = re.search(r'["\']ajax_url["\']\s*:\s*["\'](http[^"\']+)', html)
if ajax_url_match:
ajax_url = ajax_url_match.group(1).replace(r"\/", "/")
else:
ajax_url = urljoin(base_url, "/wp-admin/admin-ajax.php")
try:
res = session.post(
ajax_url,
data={"action": "manga_get_chapters", "manga": manga_id},
timeout=15,
)
if res.status_code == 200:
return res.text
except Exception:
pass
return ""
def cache_hit(url: str, force_refresh: bool = False) -> str:
if not force_refresh:
row = db_fetch_one("SELECT html_text FROM main_page WHERE url = ?", (url,))
if row is not None and row[0] is not None:
return str(row[0]) return str(row[0])
# Cache miss: fetch and save
status_code, html = robust_fetch(url) status_code, html = robust_fetch(url)
if status_code != 200: if status_code != 200:
raise RuntimeError(f"Request to {url} failed with status code {status_code}") raise RuntimeError(f"Request to {url} failed with status code {status_code}")
with sqlite3.connect(DB_PATH) as conn: extra_chapters = _fetch_wp_manga_chapters(url, html)
conn.execute( if extra_chapters:
"INSERT OR REPLACE INTO main_page (url, status_code, html_text) VALUES (?, ?, ?)", if "</body>" in html:
(url, status_code, html) html = html.replace("</body>", f"<div class='injected-wp-manga-chapters'>{extra_chapters}</div></body>")
else:
html = f"{html}\n<div class='injected-wp-manga-chapters'>{extra_chapters}</div>"
db_execute(
"INSERT INTO main_page (url, status_code, html_text) VALUES (?, ?, ?) ON CONFLICT(url) DO UPDATE SET status_code=excluded.status_code, html_text=excluded.html_text, fetched_at=CURRENT_TIMESTAMP",
(url, status_code, html),
) )
conn.commit()
return html return html
View File
+181
View File
@@ -0,0 +1,181 @@
import asyncio
import json
import os
from contextlib import asynccontextmanager
from pathlib import Path
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import HTMLResponse, FileResponse, StreamingResponse
from fastapi.staticfiles import StaticFiles
from fastapi.templating import Jinja2Templates
from .job_manager import (
jobs,
job_queues,
create_job,
cancel_job,
cancel_all_jobs,
run_job,
)
OUTPUT_DIR = Path(os.environ.get("OUTPUT_DIR", "output")).resolve()
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
@asynccontextmanager
async def lifespan(app: FastAPI):
yield
cancel_all_jobs()
app = FastAPI(title="Silksoup", lifespan=lifespan)
BASE_DIR = Path(__file__).resolve().parent
templates = Jinja2Templates(directory=str(BASE_DIR / "templates"))
if (BASE_DIR / "static").exists():
app.mount("/static", StaticFiles(directory=str(BASE_DIR / "static")), name="static")
@app.get("/", response_class=HTMLResponse)
async def index(request: Request):
return templates.TemplateResponse(request, "index.html")
@app.post("/api/jobs")
async def start_job(payload: dict):
url = (payload.get("url") or "").strip()
if not url or not (url.startswith("http://") or url.startswith("https://")):
raise HTTPException(status_code=400, detail="Valid URL required")
force_refresh = bool(payload.get("force_refresh", False))
job_id = create_job(url, force_refresh=force_refresh, output_dir=str(OUTPUT_DIR))
asyncio.create_task(run_job(job_id))
return {"job_id": job_id, "status": "queued"}
@app.post("/api/jobs/{job_id}/cancel")
async def stop_job(job_id: str):
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if job.get("status") not in ("running", "queued"):
return {"status": job.get("status"), "message": "Job is not running"}
cancel_job(job_id)
return {"status": "cancelled", "message": "Job cancelled successfully"}
@app.get("/api/jobs")
async def list_jobs():
items = []
for job_id, job in sorted(jobs.items(), key=lambda item: item[1].get("created_at", 0), reverse=True):
items.append({
"id": job["id"],
"url": job["url"],
"status": job["status"],
"stage": job["stage"],
"current": job["current"],
"total": job["total"],
"message": job["message"],
"created_at": job.get("created_at"),
"result_path": job.get("result_path"),
"error": job.get("error"),
})
return {"jobs": items[:20]}
@app.get("/api/jobs/{job_id}")
async def get_job(job_id: str):
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
return {
"id": job["id"],
"url": job["url"],
"status": job["status"],
"stage": job["stage"],
"current": job["current"],
"total": job["total"],
"message": job["message"],
"error": job["error"],
"result_path": job["result_path"],
"logs": job["logs"][-200:],
}
@app.get("/api/jobs/{job_id}/events")
async def job_events(job_id: str):
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
async def event_gen():
q = job_queues.get(job_id)
if q is None:
return
last_idx = 0
events = job.get("events", [])
for evt in events[last_idx:]:
yield f"data: {json.dumps(evt)}\n\n"
last_idx += 1
if evt.get("message") == "__DONE__":
return
while True:
if job["status"] in ("completed", "failed", "cancelled") and q.empty():
break
try:
evt = await asyncio.wait_for(q.get(), timeout=15.0)
yield f"data: {json.dumps(evt)}\n\n"
if evt.get("message") == "__DONE__":
break
except asyncio.TimeoutError:
yield ": keepalive\n\n"
if job["status"] in ("completed", "failed", "cancelled"):
break
return StreamingResponse(event_gen(), media_type="text/event-stream", headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
})
@app.get("/api/epubs")
async def list_epubs():
files = []
if OUTPUT_DIR.exists():
for p in sorted(OUTPUT_DIR.glob("*.epub"), key=lambda x: x.stat().st_mtime, reverse=True):
stat = p.stat()
files.append({
"name": p.name,
"size": stat.st_size,
"size_human": _human_size(stat.st_size),
"mtime": stat.st_mtime,
})
return {"epubs": files, "output_dir": str(OUTPUT_DIR)}
@app.get("/api/epubs/{filename}")
async def download_epub(filename: str):
safe = Path(filename).name
target = OUTPUT_DIR / safe
if not target.exists() or not target.is_file():
raise HTTPException(status_code=404, detail="File not found")
return FileResponse(str(target), media_type="application/epub+zip", filename=safe)
@app.delete("/api/epubs/{filename}")
async def delete_epub(filename: str):
safe = Path(filename).name
target = OUTPUT_DIR / safe
if not target.exists():
raise HTTPException(status_code=404, detail="File not found")
target.unlink()
return {"deleted": safe}
def _human_size(n: int) -> str:
for unit in ["B", "KB", "MB", "GB"]:
if n < 1024:
return f"{n:.1f} {unit}" if unit != "B" else f"{n} B"
n /= 1024
return f"{n:.1f} TB"
+163
View File
@@ -0,0 +1,163 @@
import asyncio
import threading
import time
import uuid
from pathlib import Path
from src.database import init_db
from src.parse import cache_hit
from src.clean_html import clean_novel_html
from src.metadata import extract_metadata_json
from src.chapter import clean_chapters
from src.epub_creator import create_epub
jobs: dict[str, dict] = {}
job_queues: dict[str, asyncio.Queue] = {}
cancel_events: dict[str, threading.Event] = {}
def _push(job_id: str, event: dict):
job = jobs.get(job_id)
if job is None:
return
job["events"].append(event)
job["updated_at"] = time.time()
if "stage" in event:
job["stage"] = event["stage"]
if "current" in event:
job["current"] = event["current"]
if "total" in event:
job["total"] = event["total"]
if "message" in event:
job["message"] = event["message"]
q = job_queues.get(job_id)
if q is not None:
try:
q.put_nowait(event)
except asyncio.QueueFull:
pass
def create_job(url: str, force_refresh: bool = False, output_dir: str = "output") -> str:
job_id = uuid.uuid4().hex[:12]
jobs[job_id] = {
"id": job_id,
"url": url,
"force_refresh": force_refresh,
"output_dir": output_dir,
"status": "queued",
"stage": "queued",
"current": 0,
"total": 0,
"message": "Queued",
"logs": [],
"events": [],
"result_path": None,
"error": None,
"created_at": time.time(),
"updated_at": time.time(),
}
job_queues[job_id] = asyncio.Queue(maxsize=1000)
cancel_events[job_id] = threading.Event()
return job_id
def cancel_job(job_id: str) -> bool:
job = jobs.get(job_id)
if not job:
return False
evt = cancel_events.get(job_id)
if evt is not None:
evt.set()
job["status"] = "cancelled"
job["stage"] = "cancelled"
job["message"] = "Process stopped by user"
job["logs"].append("[CANCEL] Process stopped by user. Cached chapters remain safe.")
_push(job_id, {"stage": "cancelled", "current": job.get("current", 0), "total": job.get("total", 0), "message": "Process stopped by user"})
_push(job_id, {"stage": "done", "current": job.get("current", 0), "total": job.get("total", 0), "message": "__DONE__"})
return True
def cancel_all_jobs():
for job_id, evt in cancel_events.items():
job = jobs.get(job_id)
if job and job.get("status") in ("running", "queued"):
evt.set()
job["status"] = "cancelled"
job["stage"] = "cancelled"
async def run_job(job_id: str):
job = jobs.get(job_id)
if job is None:
return
url = job["url"]
force_refresh = job["force_refresh"]
output_dir = job["output_dir"]
cancel_evt = cancel_events.get(job_id)
job["status"] = "running"
def _cb(stage=None, current=None, total=None, message=None, **kwargs):
evt = {"stage": stage or "unknown", "current": current or 0, "total": total or 0, "message": message or ""}
_push(job_id, evt)
if message:
job["logs"].append(f"[{stage}] {message}")
try:
if cancel_evt and cancel_evt.is_set():
raise InterruptedError("Cancelled before start")
init_db()
_push(job_id, {"stage": "fetching", "current": 0, "total": 1, "message": "Fetching index page..."})
job["logs"].append(f"Fetching {url}")
html = await asyncio.to_thread(cache_hit, url, force_refresh)
if cancel_evt and cancel_evt.is_set():
raise InterruptedError("Cancelled after fetch")
cleaned = clean_novel_html(html)
_push(job_id, {"stage": "metadata", "current": 0, "total": 1, "message": "Extracting metadata via LLM..."})
job["logs"].append("Extracting metadata")
metadata = await asyncio.to_thread(extract_metadata_json, cleaned, url, force_refresh)
metadata["novel_url"] = url
if cancel_evt and cancel_evt.is_set():
raise InterruptedError("Cancelled after metadata")
volumes = metadata.get("volumes", [])
total_chapters = sum(len(v.get("chapters", [])) for v in volumes)
_push(job_id, {"stage": "cleaning", "current": 0, "total": total_chapters, "message": f"Cleaning {total_chapters} chapters"})
job["logs"].append(f"Found {len(volumes)} volumes, {total_chapters} chapters")
await asyncio.to_thread(clean_chapters, metadata, force_refresh, _cb, cancel_evt)
if cancel_evt and cancel_evt.is_set():
raise InterruptedError("Cancelled before EPUB building")
_push(job_id, {"stage": "epub", "current": 0, "total": total_chapters, "message": "Building EPUB..."})
output_path = await asyncio.to_thread(create_epub, metadata, output_dir, _cb, cancel_evt)
job["result_path"] = output_path
job["status"] = "completed"
job["stage"] = "completed"
job["message"] = f"Completed: {Path(output_path).name}"
job["logs"].append(f"Generated {output_path}")
_push(job_id, {"stage": "completed", "current": total_chapters, "total": total_chapters, "message": job["message"]})
_push(job_id, {"stage": "done", "current": total_chapters, "total": total_chapters, "message": "__DONE__"})
except InterruptedError:
job["status"] = "cancelled"
job["stage"] = "cancelled"
job["message"] = "Process stopped by user"
job["logs"].append("[CANCEL] Process stopped by user. Cached chapters remain safe.")
_push(job_id, {"stage": "cancelled", "current": job.get("current", 0), "total": job.get("total", 0), "message": "Process stopped by user"})
_push(job_id, {"stage": "done", "current": job.get("current", 0), "total": job.get("total", 0), "message": "__DONE__"})
except Exception as e:
job["status"] = "failed"
job["stage"] = "failed"
job["error"] = str(e)
job["message"] = str(e)
job["logs"].append(f"[ERROR] {e}")
_push(job_id, {"stage": "failed", "current": job.get("current", 0), "total": job.get("total", 0), "message": str(e)})
_push(job_id, {"stage": "done", "current": 0, "total": 0, "message": "__DONE__"})
+541
View File
@@ -0,0 +1,541 @@
<!DOCTYPE html>
<html lang="en" class="dark">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Silksoup — Novel to EPUB</title>
<script src="https://cdn.tailwindcss.com"></script>
<script>
tailwind.config = { darkMode: 'class', theme: { extend: { fontFamily: { sans: ['Inter','system-ui','sans-serif'] } } } }
</script>
<style>
::-webkit-scrollbar{width:6px;height:6px}::-webkit-scrollbar-thumb{background:#27272a;border-radius:999px}::-webkit-scrollbar-track{background:transparent}
@keyframes pulseGlow { 0%, 100% { opacity: 0.4; } 50% { opacity: 1; } }
.animate-glow { animation: pulseGlow 2s cubic-bezier(0.4, 0, 0.6, 1) infinite; }
</style>
</head>
<body class="bg-[#09090b] text-zinc-100 min-h-screen flex flex-col antialiased selection:bg-zinc-800">
<header class="border-b border-zinc-800/80 bg-zinc-950/80 backdrop-blur sticky top-0 z-30">
<div class="max-w-4xl mx-auto px-4 py-3.5 flex items-center justify-between">
<div class="flex items-center gap-3">
<div class="w-8 h-8 rounded-lg bg-white text-black font-black grid place-items-center text-sm shadow-sm">S</div>
<div>
<h1 class="font-semibold tracking-tight text-sm leading-none flex items-center gap-2">
Silksoup
<span id="headerBadge" class="hidden text-[10px] font-medium px-2 py-0.5 rounded-full bg-emerald-500/10 text-emerald-400 border border-emerald-500/20">Active</span>
</h1>
<p class="text-[11px] text-zinc-400 mt-0.5">Novel to EPUB pipeline</p>
</div>
</div>
<div class="flex items-center gap-3">
<button id="notifyBtn" onclick="toggleNotifications()" title="Enable Desktop Notifications" class="text-xs text-zinc-400 hover:text-zinc-200 px-2.5 py-1.5 rounded-lg border border-zinc-800 hover:border-zinc-700 transition flex items-center gap-1.5">
<span id="notifyIcon">🔔</span>
<span id="notifyText" class="hidden sm:inline">Notify</span>
</button>
<a href="/api/epubs" target="_blank" class="text-xs text-zinc-400 hover:text-zinc-200 px-2.5 py-1.5 rounded-lg border border-zinc-800 hover:border-zinc-700 transition">API</a>
</div>
</div>
</header>
<main class="max-w-4xl mx-auto w-full px-4 py-6 flex flex-col gap-6 flex-1">
<!-- New Job Card -->
<section class="rounded-2xl border border-zinc-800/80 bg-zinc-900/40 p-5 shadow-sm">
<div class="flex items-center justify-between mb-3">
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Scrape New Novel</h2>
<button onclick="pasteClipboard()" class="text-[11px] text-zinc-400 hover:text-white flex items-center gap-1 transition">
<span>📋</span> Paste URL
</button>
</div>
<form id="jobForm" class="space-y-4" onsubmit="return false">
<div class="relative">
<input id="urlInput" type="url" required placeholder="https://kakuyomu.jp/works/... or novel index URL" class="w-full rounded-xl bg-zinc-950 border border-zinc-800 px-4 py-3.5 text-sm placeholder:text-zinc-600 focus:outline-none focus:border-zinc-500 transition shadow-inner">
</div>
<div class="flex flex-col sm:flex-row sm:items-center justify-between gap-3 pt-1">
<label class="flex items-center gap-2 text-xs text-zinc-400 cursor-pointer select-none">
<input id="refreshInput" type="checkbox" class="rounded border-zinc-700 bg-zinc-950 text-white focus:ring-0 focus:ring-offset-0">
Force refresh (bypass database cache)
</label>
<button id="startBtn" class="sm:w-auto w-full rounded-xl bg-white text-black text-sm font-semibold px-7 py-3 hover:bg-zinc-200 active:scale-[0.99] transition shrink-0 flex items-center justify-center gap-2">
<span id="btnSpinner" class="hidden inline-block w-4 h-4 border-2 border-black border-t-transparent rounded-full animate-spin"></span>
<span id="btnText">Start Scraping → EPUB</span>
</button>
</div>
</form>
</section>
<!-- Active / In-Progress Job Section -->
<section id="activeJob" class="hidden rounded-2xl border border-zinc-800 bg-zinc-900/60 p-5 shadow-sm transition-all">
<div class="flex items-center justify-between mb-3">
<div class="flex items-center gap-2">
<span id="statusPulse" class="w-2.5 h-2.5 rounded-full bg-emerald-400 animate-glow"></span>
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Task Monitor</h2>
</div>
<div class="flex items-center gap-2">
<button id="cancelJobBtn" onclick="requestCancelJob()" class="text-[11px] px-2.5 py-0.5 rounded-full border border-rose-900/60 bg-rose-950/40 text-rose-300 hover:bg-rose-900/60 transition">Stop Process</button>
<span id="jobStatusBadge" class="text-[11px] font-mono uppercase px-2.5 py-0.5 rounded-full bg-zinc-800 text-zinc-300 border border-zinc-700">idle</span>
</div>
</div>
<div class="space-y-3.5">
<div class="flex items-center justify-between text-xs gap-3">
<div class="flex items-center gap-2 min-w-0">
<span id="jobStage" class="px-2 py-0.5 rounded bg-zinc-800/80 text-zinc-300 font-mono text-[10px] uppercase tracking-wider">init</span>
<span id="jobMessage" class="text-zinc-300 truncate font-medium">Starting...</span>
</div>
<span id="jobCounter" class="text-zinc-400 tabular-nums font-mono font-medium shrink-0">0 / 0</span>
</div>
<div class="h-2 rounded-full bg-zinc-950 border border-zinc-800 overflow-hidden relative">
<div id="jobProgress" class="h-full bg-white transition-all duration-300" style="width:0%"></div>
</div>
<!-- Log Toolbar -->
<div class="flex items-center justify-between text-[11px] text-zinc-500 px-0.5 pt-1">
<span>Execution Log</span>
<div class="flex items-center gap-3">
<label class="flex items-center gap-1.5 cursor-pointer select-none hover:text-zinc-300">
<input id="autoScrollCheck" type="checkbox" checked class="rounded border-zinc-700 bg-zinc-950 text-white focus:ring-0">
Auto-scroll
</label>
<button onclick="clearLogs()" class="hover:text-zinc-300">Clear</button>
<button onclick="toggleLogHeight()" id="expandLogBtn" class="hover:text-zinc-300">Expand</button>
</div>
</div>
<div id="jobLog" class="h-44 overflow-auto rounded-xl bg-zinc-950 border border-zinc-800/90 p-3.5 font-mono text-[11px] leading-relaxed text-zinc-400 whitespace-pre-wrap transition-all"></div>
<div id="jobResult" class="hidden pt-1 flex items-center gap-3">
<a id="downloadLink" href="#" class="inline-flex items-center gap-2 rounded-xl bg-white text-black text-xs font-semibold px-5 py-2.5 hover:bg-zinc-200 transition shadow-sm">
<span>📥</span> Download EPUB
</a>
<button onclick="dismissActiveJob()" class="text-xs text-zinc-400 hover:text-zinc-200 px-3 py-2 rounded-xl border border-zinc-800 hover:bg-zinc-800/50 transition">Dismiss</button>
</div>
</div>
</section>
<!-- Library Section -->
<section class="rounded-2xl border border-zinc-800/80 bg-zinc-900/40 p-5 shadow-sm">
<div class="flex flex-col sm:flex-row sm:items-center justify-between gap-3 mb-4">
<div class="flex items-center gap-2">
<h2 class="text-xs font-semibold uppercase tracking-wider text-zinc-400">Library</h2>
<span id="epubCount" class="text-[11px] text-zinc-500 font-mono"></span>
</div>
<div class="flex items-center gap-2">
<div class="relative flex-1 sm:w-60">
<input id="searchEpub" type="text" oninput="filterEpubs()" placeholder="Filter books..." class="w-full rounded-xl bg-zinc-950 border border-zinc-800 px-3 py-1.5 text-xs placeholder:text-zinc-600 focus:outline-none focus:border-zinc-600">
</div>
<button onclick="loadEpubs()" class="text-xs px-3 py-1.5 rounded-xl border border-zinc-800 hover:bg-zinc-800 hover:text-white text-zinc-400 transition shrink-0">Refresh</button>
</div>
</div>
<div id="epubList" class="space-y-2.5"></div>
<div id="epubEmpty" class="text-xs text-zinc-500 py-8 text-center hidden">
<div class="text-2xl mb-1">📚</div>
No EPUBs found. Scrape a novel to populate your library.
</div>
</section>
<!-- How it works Footer -->
<section class="rounded-2xl border border-zinc-800/60 bg-zinc-900/20 p-4">
<h3 class="text-xs font-semibold text-zinc-400 mb-2">Workflow Pipeline</h3>
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-2 text-[11px] text-zinc-500 leading-normal">
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
<span class="font-semibold text-zinc-400 block mb-0.5">1. Ingest & Cache</span>
Fetches novel HTML via anti-bot curl_cffi into Postgres/SQLite.
</div>
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
<span class="font-semibold text-zinc-400 block mb-0.5">2. Parse Index</span>
DeepSeek LLM extracts chronological volumes and chapter URLs.
</div>
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
<span class="font-semibold text-zinc-400 block mb-0.5">3. Sanitize Content</span>
Cleans site bloat, keeps translator notes, wraps semantic HTML.
</div>
<div class="p-2.5 rounded-xl bg-zinc-950/40 border border-zinc-800/50">
<span class="font-semibold text-zinc-400 block mb-0.5">4. Build EPUB</span>
Downloads images with referers, compresses WEBP/JPG, compiles book.
</div>
</div>
</section>
</main>
<!-- In-Page Toast Container -->
<div id="toastContainer" class="fixed bottom-4 right-4 z-50 flex flex-col gap-2 pointer-events-none"></div>
<footer class="border-t border-zinc-800/80 py-4 text-center text-[11px] text-zinc-600">
Silksoup — Self-hosted EPUB Generator • Fast, headless, and cached.
</footer>
<script>
const $ = s => document.querySelector(s);
let currentJobId = localStorage.getItem('silksoup_active_job') || null;
let evtSource = null;
let allEpubs = [];
let originalTitle = document.title;
let notificationsAllowed = Notification.permission === 'granted';
updateNotifyButton();
function updateNotifyButton(){
if (Notification.permission === 'granted') {
$('#notifyBtn').classList.add('text-emerald-400', 'border-emerald-500/30');
$('#notifyText').textContent = 'Notifications on';
}
}
async function toggleNotifications(){
if (Notification.permission === 'default') {
const res = await Notification.requestPermission();
if (res === 'granted') {
showToast('Notifications enabled!', 'success');
updateNotifyButton();
}
} else if (Notification.permission === 'denied') {
showToast('Notifications are blocked in your browser settings.', 'error');
} else {
showToast('Notifications are already enabled.', 'info');
}
}
function notify(title, body){
if (Notification.permission === 'granted' && document.hidden) {
new Notification(title, { body, icon: '/static/favicon.ico' });
}
}
function showToast(message, type = 'info'){
const container = $('#toastContainer');
const toast = document.createElement('div');
const bg = type === 'error' ? 'bg-rose-950/90 border-rose-800 text-rose-200' :
type === 'success' ? 'bg-emerald-950/90 border-emerald-800 text-emerald-200' :
'bg-zinc-900/90 border-zinc-800 text-zinc-200';
toast.className = `pointer-events-auto rounded-xl border px-4 py-3 text-xs shadow-lg backdrop-blur flex items-center gap-2 transition-all transform translate-y-2 opacity-0 duration-200 ${bg}`;
toast.innerHTML = `<span>${type === 'error' ? '⚠️' : type === 'success' ? '✓' : 'ℹ️'}</span><span class="flex-1">${message}</span>`;
container.appendChild(toast);
requestAnimationFrame(() => {
toast.classList.remove('translate-y-2', 'opacity-0');
});
setTimeout(() => {
toast.classList.add('opacity-0', 'translate-y-2');
setTimeout(() => toast.remove(), 250);
}, 4000);
}
async function pasteClipboard(){
try {
const text = await navigator.clipboard.readText();
if (text) {
$('#urlInput').value = text.trim();
showToast('Pasted URL from clipboard', 'info');
}
} catch(e) {
showToast('Clipboard access denied', 'error');
}
}
function setBtnLoading(loading){
const btn = $('#startBtn');
const spinner = $('#btnSpinner');
const text = $('#btnText');
if (loading) {
btn.disabled = true;
spinner.classList.remove('hidden');
text.textContent = 'Submitting...';
} else {
btn.disabled = false;
spinner.classList.add('hidden');
text.textContent = 'Start Scraping → EPUB';
}
}
$('#startBtn').addEventListener('click', async () => {
const url = $('#urlInput').value.trim();
if (!url) {
showToast('Please enter a novel URL', 'error');
return;
}
setBtnLoading(true);
try {
const res = await fetch('/api/jobs', {
method: 'POST',
headers: {'Content-Type':'application/json'},
body: JSON.stringify({ url, force_refresh: $('#refreshInput').checked })
});
if (!res.ok) {
const j = await res.json();
throw new Error(j.detail || 'Failed to submit job');
}
const data = await res.json();
currentJobId = data.job_id;
localStorage.setItem('silksoup_active_job', currentJobId);
showActive();
connectSSE(currentJobId);
showToast('Job started successfully', 'success');
} catch(e) {
showToast(e.message, 'error');
} finally {
setBtnLoading(false);
}
});
function showActive(){
$('#activeJob').classList.remove('hidden');
$('#headerBadge').classList.remove('hidden');
$('#cancelJobBtn').classList.remove('hidden');
$('#jobLog').textContent = '';
$('#jobProgress').style.width = '0%';
$('#jobCounter').textContent = '0 / 0';
$('#jobStage').textContent = 'queued';
$('#jobStatusBadge').textContent = 'running';
$('#jobResult').classList.add('hidden');
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400 animate-glow';
$('#activeJob').scrollIntoView({ behavior: 'smooth', block: 'nearest' });
}
async function requestCancelJob(){
if (!currentJobId) return;
const btn = $('#cancelJobBtn');
btn.disabled = true;
btn.textContent = 'Stopping...';
try {
const res = await fetch(`/api/jobs/${currentJobId}/cancel`, { method: 'POST' });
if (res.ok) {
showToast('Process cancellation requested. Cached chapters remain safe.', 'info');
}
} catch(e) {
showToast('Failed to cancel job', 'error');
} finally {
btn.disabled = false;
btn.textContent = 'Stop Process';
}
}
function dismissActiveJob(){
localStorage.removeItem('silksoup_active_job');
currentJobId = null;
if (evtSource) evtSource.close();
$('#activeJob').classList.add('hidden');
$('#headerBadge').classList.add('hidden');
document.title = originalTitle;
}
function connectSSE(jobId){
if (evtSource) evtSource.close();
evtSource = new EventSource(`/api/jobs/${jobId}/events`);
evtSource.onmessage = (e) => {
if (e.data.startsWith(':')) return;
const evt = JSON.parse(e.data);
if (evt.message === '__DONE__') {
evtSource.close();
refreshJob();
loadEpubs();
return;
}
const pct = evt.total ? Math.round(evt.current / evt.total * 100) : 0;
$('#jobStage').textContent = evt.stage;
$('#jobMessage').textContent = evt.message || '';
$('#jobCounter').textContent = `${evt.current} / ${evt.total}`;
$('#jobProgress').style.width = pct + '%';
if (evt.stage === 'scraping' || evt.stage === 'cleaning' || evt.stage === 'epub') {
document.title = `(${pct}%) Silksoup`;
}
if (evt.stage === 'failed') {
$('#jobStatusBadge').textContent = 'failed';
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
$('#cancelJobBtn').classList.add('hidden');
document.title = `❌ Failed — Silksoup`;
showToast(`Job failed: ${evt.message}`, 'error');
}
if (evt.stage === 'cancelled') {
$('#jobStatusBadge').textContent = 'cancelled';
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
$('#cancelJobBtn').classList.add('hidden');
document.title = `Silksoup`;
showToast('Scraping stopped. Completed chapters are cached in database.', 'info');
}
if (evt.stage === 'completed') {
$('#jobStatusBadge').textContent = 'completed';
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400';
$('#cancelJobBtn').classList.add('hidden');
document.title = `✓ Completed — Silksoup`;
notify('Silksoup EPUB Ready!', evt.message || 'Your book has been compiled successfully.');
showToast('EPUB compiled successfully!', 'success');
}
if (evt.message && evt.message !== '__DONE__') {
const log = $('#jobLog');
log.textContent += `[${evt.stage}] ${evt.message}\n`;
if ($('#autoScrollCheck').checked) {
log.scrollTop = log.scrollHeight;
}
}
};
evtSource.onerror = () => {
// Retry handled by browser EventSource
};
}
async function refreshJob(){
if (!currentJobId) return;
const res = await fetch(`/api/jobs/${currentJobId}`);
if (!res.ok) return;
const j = await res.json();
$('#jobStatusBadge').textContent = j.status;
if (j.status === 'completed') {
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-emerald-400';
$('#cancelJobBtn').classList.add('hidden');
if (j.result_path) {
const name = j.result_path.split(/[\\/]/).pop();
$('#downloadLink').href = `/api/epubs/${encodeURIComponent(name)}`;
$('#downloadLink').innerHTML = `<span>📥</span> Download ${name}`;
$('#jobResult').classList.remove('hidden');
}
}
if (j.status === 'cancelled') {
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-amber-500';
$('#cancelJobBtn').classList.add('hidden');
}
if (j.status === 'failed') {
$('#statusPulse').className = 'w-2.5 h-2.5 rounded-full bg-rose-500';
$('#cancelJobBtn').classList.add('hidden');
$('#jobLog').textContent += `\n[ERROR] ${j.error || j.message}\n`;
}
}
function clearLogs(){
$('#jobLog').textContent = '';
}
let isLogExpanded = false;
function toggleLogHeight(){
const log = $('#jobLog');
const btn = $('#expandLogBtn');
if (isLogExpanded) {
log.classList.remove('h-96');
log.classList.add('h-44');
btn.textContent = 'Expand';
isLogExpanded = false;
} else {
log.classList.remove('h-44');
log.classList.add('h-96');
btn.textContent = 'Collapse';
isLogExpanded = true;
}
}
async function loadEpubs(){
try {
const res = await fetch('/api/epubs');
const data = await res.json();
allEpubs = data.epubs || [];
$('#epubCount').textContent = allEpubs.length ? `(${allEpubs.length})` : '';
renderEpubs(allEpubs);
} catch(e) {
showToast('Failed to load library', 'error');
}
}
function filterEpubs(){
const query = $('#searchEpub').value.toLowerCase().trim();
if (!query) {
renderEpubs(allEpubs);
return;
}
const filtered = allEpubs.filter(f => f.name.toLowerCase().includes(query));
renderEpubs(filtered);
}
function renderEpubs(list){
const container = $('#epubList');
container.innerHTML = '';
if (!list.length) {
$('#epubEmpty').classList.remove('hidden');
return;
}
$('#epubEmpty').classList.add('hidden');
for (const f of list) {
const row = document.createElement('div');
row.className = 'group flex items-center gap-3.5 rounded-xl border border-zinc-800/80 bg-zinc-950/80 px-4 py-3 hover:border-zinc-700 transition';
// Clean filename for nice display
const cleanTitle = f.name.replace(/\.epub$/i, '').replace(/[-_]/g, ' ');
row.innerHTML = `
<div class="w-8 h-8 rounded-lg bg-zinc-900 border border-zinc-800 flex items-center justify-center text-xs shrink-0 text-zinc-400 group-hover:text-white transition">📖</div>
<div class="flex-1 min-w-0">
<div class="truncate text-zinc-200 text-xs font-medium capitalize" title="${cleanTitle}">${cleanTitle}</div>
<div class="text-[11px] text-zinc-400 font-mono mt-0.5">${f.size_human} • ${new Date(f.mtime*1000).toLocaleDateString()} ${new Date(f.mtime*1000).toLocaleTimeString([], {hour: '2-digit', minute:'2-digit'})}</div>
</div>
<div class="flex items-center gap-2 shrink-0">
<a href="/api/epubs/${encodeURIComponent(f.name)}" download class="text-xs px-3.5 py-1.5 rounded-xl bg-white text-black font-semibold hover:bg-zinc-200 transition">Download</a>
<div class="relative inline-block" id="delWrap_${CSS.escape(f.name)}">
<button onclick="confirmDelete('${encodeURIComponent(f.name)}')" class="text-xs px-2.5 py-1.5 rounded-xl border border-zinc-800 hover:border-rose-900/60 hover:text-rose-400 text-zinc-400 transition">Delete</button>
</div>
</div>
`;
container.appendChild(row);
}
}
function confirmDelete(encodedName){
const name = decodeURIComponent(encodedName);
const wrap = document.getElementById(`delWrap_${CSS.escape(name)}`);
if (!wrap) return;
wrap.innerHTML = `
<div class="flex items-center gap-1.5 bg-rose-950/60 border border-rose-800/80 rounded-xl p-0.5">
<button onclick="executeDelete('${encodeURIComponent(name)}')" class="text-[11px] px-2 py-1 bg-rose-600 text-white rounded-lg font-medium hover:bg-rose-500">Confirm</button>
<button onclick="loadEpubs()" class="text-[11px] px-2 py-1 text-zinc-400 hover:text-white">Cancel</button>
</div>
`;
}
async function executeDelete(encodedName){
try {
const res = await fetch(`/api/epubs/${encodedName}`, { method: 'DELETE' });
if (!res.ok) throw new Error('Delete failed');
showToast('EPUB removed from library', 'info');
loadEpubs();
} catch(e) {
showToast('Failed to delete file', 'error');
}
}
// Auto-restore active job on load if any
async function checkStoredJob(){
if (!currentJobId) return;
try {
const res = await fetch(`/api/jobs/${currentJobId}`);
if (res.ok) {
const j = await res.json();
showActive();
$('#jobStage').textContent = j.stage;
$('#jobMessage').textContent = j.message || '';
$('#jobCounter').textContent = `${j.current} / ${j.total}`;
const pct = j.total ? Math.round(j.current / j.total * 100) : 0;
$('#jobProgress').style.width = pct + '%';
if (j.logs) {
$('#jobLog').textContent = j.logs.join('\n') + '\n';
}
if (j.status === 'running' || j.status === 'queued') {
connectSSE(currentJobId);
} else {
refreshJob();
}
} else {
localStorage.removeItem('silksoup_active_job');
}
} catch(e) {
localStorage.removeItem('silksoup_active_job');
}
}
checkStoredJob();
loadEpubs();
</script>
</body>
</html>