Welcome to the Wiki.# LotusGen Documentation & Wiki
LotusGen is a high-throughput, low-memory dummy and synthetic data generation platform designed for software engineers, data analysts, and QA teams. It enables the creation of complex, relational, and reproducible tabular datasets at scale in CSV and Apache Parquet formats.
1. System Overview & Architecture
LotusGen combines a reactive TypeScript frontend with a high-performance Python generation engine:
- Frontend: React 19, TypeScript, Vite, Tailwind CSS v4, Phosphor Icons.
- Backend API: FastAPI with SQLite metadata storage.
- Streaming Core: Polars and PyArrow chunked pipeline for bounded memory consumption during large-scale generation.
- Safety Engine: Custom AST (Abstract Syntax Tree) expression sandbox for evaluating custom formulas without unsafe
eval()calls. - Dependency Resolver: Intra-table and inter-table DAG (Directed Acyclic Graph) validation to prevent cyclical joins and enforce topological ordering.
2. Core Concepts & Table Archetypes
LotusGen models data relationships using four distinct table archetypes:
Record (record)
- Represents transactional, event, or entity data (e.g.,
orders,user_events,payments). - Supports randomized distributions, reference lookups, regex masks, and custom formulas.
- Configurable row limits (from hundreds to millions of records).
Aggregate (aggregate)
- Summary tables designed to roll up metric distributions or aggregated scenarios.
- Allows defining calculated aggregate fields based on relational dependencies.
Reference (reference)
- Dimension tables acting as master catalog sources (e.g.,
products,categories,zip_codes). - Edited via the built-in in-browser spreadsheet interface.
- Referenced by Record tables using
VLOOKUP-style key-value column joins.
Collection (collection)
- Weighted categorical choice pools.
- Allows defining custom discrete distributions (e.g.,
Status: [("active", 0.7), ("pending", 0.2), ("banned", 0.1)]). - Usable across any record field set to the
collectiontype.
3. Supported Column Types & Roles
Column Roles
randomized: Directly sampled from a statistical distribution, regex pattern, or built-in generator.function: Evaluated dynamically row-by-row using an AST-sandboxed formula referencing upstream columns.reference: Performs an index lookup against areferencetable matching a source key column.static: Outputs a constant fixed literal value for every generated row.
Column Data Types
| Data Type | Description | Configuration Options |
|---|---|---|
| string | Text values, regex patterns | Pattern mask, prefix, suffix |
| integer | Whole numeric values | Min, max, distribution type |
| float | Floating point numbers | Min, max, decimal precision |
| datetime | ISO timestamps / dates | Start date, end date, offset days |
| boolean | True / False boolean flags | Probability true (0.0 to 1.0) |
| uuid7 | Time-sortable UUIDv7 IDs | Deterministic from project seed |
| choice | Inlined discrete categorical list | List of values, weights |
| collection | Reference to a Collection table | Target collection name |
4. Formula Expression Engine
LotusGen allows custom column derivations and conditional filtering through a secure AST sandbox.
Intra-Table Dependencies
A function column can reference any column defined above it in the table schema. The intra-table DAG automatically validates column evaluation order.
Supported Functions & Syntax
- Math Operators:
+,-,, `/`, `//`, `%`, - Comparisons:*
==,!=,<,<=,>,>= - Logical:
and,or,not - Standard Math:
abs(x),round(x, n),min(a, b),max(a, b),sqrt(x) - Conditionals: Python ternary expressions:
value_if_true if condition else value_if_false - Examples:
- Price with Tax:
round(unit_price * (1 + tax_rate), 2) - Customer Tier:
'VIP' if total_spend > 1000 else 'Regular' - Full Name:
first_name + ' ' + last_name
- Price with Tax:
Conditional Filtering (exclude_if)
Tables support an optional exclude_if formula. If the condition evaluates to True, the row is discarded from the output stream.
5. Output Formats & Determinism
Reproducibility via Seeds
Every project defines a master integer seed. All pseudorandom generation (including UUIDv7 generation, weighted random sampling, and regex token generation) is strictly deterministic. Re-running a generation job with the same seed yields identical row sequences.
Formats
- CSV: Standard comma-separated output with streaming header and row chunks.
- Apache Parquet: Columnar storage format with Snappy compression for minimal disk footprint and analytical workflows.
6. Installation & Self-Hosting
Quick Start with Docker Compose
git clone https://git.sylphian.my.id/Sylphian/lotusgen.git
cd lotusgen
docker compose up -d --build
The service will start on port 8000.
Environment Variables
| Variable | Default | Purpose |
|---|---|---|
LOTUSGEN_DB_PATH |
/app/data/lotusgen.db |
Path to the SQLite metadata database |
LOTUSGEN_ARTIFACTS_DIR |
/app/artifacts |
Output directory for generated CSV/Parquet files |
Local Development Setup
- Backend (FastAPI):
cd backend
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn backend.main:app --reload --port 8000
- Frontend (Vite + React):
cd frontend
npm install
npm run dev
7. API Reference Summary
LotusGen exposes a REST API for automated generation and integration testing:
| Method | Endpoint | Description |
|---|---|---|
 |
/api/health |
Service health check |
|
/api/projects |
List all data generation projects |
 |
/api/projects |
Create a new project schema |
|
/api/projects/{id} |
Retrieve full project schema and dependencies |
|
/api/projects/{id}[/tables](tg://bot_command?command=tables) |
Add a table schema to a project |
|
/api/tables/{id}[/dry](tg://bot_command?command=dry)-run |
Sample a live preview of generated rows |
|
/api/tables/{id}[/generate](tg://bot_command?command=generate) |
Start asynchronous generation job |
|
/api/jobs/{id} |
Check job metrics (rows/sec, file size) |
|
/api/jobs/{id}[/download](tg://bot_command?command=download) |
Download generated CSV or Parquet artifact |