Silksoup scrapes web novel index pages, uses an OpenAI-compatible LLM to detect volumes and chapters, cleans out site ads and junk, downloads illustrations, and packages everything into an offline .epub file ready for your e-reader, phone, or tablet.

Key Features

Dual Modes: Use the visual Web Dashboard or the terminal CLI tool—both share the same engine.
Smart Chapter Extraction: Uses AI to understand complex table of contents, volumes, and reading order.
Clean Reading Experience: Strips navigation links, donation banners, and website watermarks while preserving important translator notes, formatting, and character dialogue.
Image & Illustration Support: Downloads cover art and embedded chapter illustrations (supports WEBP, JPEG, PNG, GIF) with anti-hotlinking headers.
Smart Caching (PostgreSQL & SQLite): Stores downloaded chapters so re-runs and updates don't re-download what you already have.
Self-Hostable & Docker Ready: Run it locally or host it on a VPS behind WireGuard / VPN.

Quick Start (Web App)

Option A: Using Docker (Recommended for Servers / VPS)

Clone the repository:

git clone https://github.com/your-username/silksoup.git
cd silksoup

Set up your environment variables:

cp .env.example .env

Open .env and configure your OpenAI-compatible LLM:

LLM_API_KEY=your_api_key_here
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=ds/deepseek-v4-flash

# Optional: Connect to your PostgreSQL database (leave blank to use local SQLite)
DATABASE_URL=postgresql://user:password@localhost:5432/silksoup

Start the service:
```
docker compose up -d --build
```
Open your browser: Go to http://localhost:8000 (or http://YOUR_SERVER_IP:8000).

Option B: Running Locally with Python

Install dependencies:

pip install -r requirements.txt
# or with uv:
uv pip install -r requirements.txt

Configure .env (see step 2 above).
Start the web server:
```
python server.py
```
Open http://localhost:8000 in your browser.

Web Dashboard Guide

Enter Novel URL: Paste the main index or table-of-contents link of the novel into the input box.
Start Scraping: Click "Start Scraping -> EPUB".
Live Progress: Watch real-time progress as chapters are fetched, cleaned, and compiled into the EPUB.
Download & Manage: Once complete, click "Download EPUB" or manage your compiled books directly from the built-in Library.

CLI Usage

If you prefer working from the terminal, run main.py directly:

CLI Syntax:

python main.py <url> [-o OUTPUT] [--refresh [{none,llm,all}]]

Flags & Options:

Option / Flag	Type	Description
`url`	Positional (Required)	The main index / table-of-contents URL of the novel.
`-o`, `--output`	Optional	Output directory for the compiled EPUB file (default: `./output`).
`--refresh`	Optional (Choice)	Cache refresh mode: `none` (default, full cache), `llm` (reuse HTML, re-run AI), or `all` (full scrape anew). Providing `--refresh` without an argument defaults to `all`.
`-h`, `--help`	Optional (Flag)	Show command-line help message and exit.

Where Are My EPUB Files?

All compiled .epub files are saved directly inside the output/ directory in the project folder:

Host Path: ./output/<novel-title>.epub
Docker Path: /app/output/<novel-title>.epub

You can move these files to your e-reader (Kindle, Kobo, Boox) or integrate the folder with media servers like Calibre-Web, Kavita, or Audiobookshelf.

Configuration Reference

All settings can be configured inside your .env file:

Variable	Description	Default
`LLM_API_KEY`	Required. Your API key for the OpenAI-compatible LLM.	None
`LLM_BASE_URL`	Required. Base URL of any OpenAI-compatible API endpoint.	None
`LLM_MODEL`	Required. Model identifier used for extraction & cleaning.	None
`DATABASE_URL`	PostgreSQL connection string. If omitted, uses local `sqlite.db`.	None (SQLite)
`OUTPUT_DIR`	Directory where compiled EPUBs are stored.	`output`