DevMort/cl-webnovel-to-org
Watch
1
0
Fork
You've already forked cl-webnovel-to-org
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
20 commits 2 branches 0 tags 361 KiB
  • Common Lisp 97%
  • Makefile 3%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ryan Joshua fe69b93475 docs: update README with full dependency list and new features 
Corrects the license in the .asd file from AGPLV3 to MIT to match the
LICENSE file.

Updates the README.md to include the full list of dependencies from the
.asd file, and adds a new 'Advanced Features' section to document the
multiple strategies used to find the next chapter link.
2025-07-09 10:43:10 +08:00
cl-webnovel-to-org.asd docs: update README with full dependency list and new features 2025-07-09 10:43:10 +08:00
cl-webnovel-to-org.lisp add random delays to look like a human, for rate limits 2025-07-09 10:38:21 +08:00
LICENSE updated readme and license 2025-07-09 08:26:20 +08:00
logo.png logo 2025-07-09 08:43:58 +08:00
Makefile 🥀 uploaded 2024-04-14 18:53:24 +08:00
package.lisp 🥀 uploaded 2024-04-14 18:53:24 +08:00
README.md docs: update README with full dependency list and new features 2025-07-09 10:43:10 +08:00

README.md

Common Lisp Logo

cl-webnovel-to-org

Language: Common Lisp License: MIT

A command-line tool written in Common Lisp to scrape webnovels from a website and compile them into a single, clean Org-mode file. Once you have the Org file, you can easily convert it to other formats like EPUB or PDF using Pandoc.

Features

  • Interactive Setup: Guides you through the process of providing the necessary information.
  • Scriptable: Can be run non-interactively by providing arguments, making it suitable for scripts.
  • Content Scraping: Extracts chapter titles and body content.
  • Image Support: Preserves images from the novel.
  • Comment Filtering: Can exclude comment sections from the output.
  • Custom Stop Points: Allows you to specify text that, when encountered, will stop the scraping process (e.g., "End of Volume").
  • Org-mode Formatting: Generates a well-structured Org file with a title, author, and creation date.

Dependencies

  • Dexador: High-performance HTTP client for Common Lisp.
  • lquery: A DOM manipulation library for Common Lisp, similar to jQuery.
  • local-time: A library for manipulating dates and times.
  • split-sequence: A utility for splitting sequences.
  • quri: A URI manipulation library.
  • jonathan: A JSON encoder/decoder.
  • cl-ppcre: A portable Perl-compatible regular expression library.

These will be installed automatically via Quicklisp.

Advanced Features

This tool includes several strategies to find the "next chapter" link, making it more robust and adaptable to different website structures. These strategies are tried in order until one succeeds:

  1. User-Provided Selector: Uses the CSS selector you provide.
  2. Keyword Search: Looks for links with common "next" keywords (e.g., "Next", "next", ">").
  3. rel="next" Attribute: Searches for a link with the rel="next" attribute, a common web standard.
  4. Embedded JSON Data: Parses embedded JSON data in the page, which is common on modern, JavaScript-heavy websites.

Installation

There are two ways to install and use this tool.

1. From Source (Recommended)

  1. Clone this repository into your ~/quicklisp/local-projects/ directory.
  2. Ensure you have a modern Common Lisp implementation like SBCL installed.
  3. You can then run the tool directly from your Lisp REPL.

2. Build a Binary

The included Makefile allows you to build a standalone executable.

# Build the binary
make build

# Install the binary to ~/.local/bin/
make install

This will create a cl-webnovel-to-org executable that you can run from anywhere.

Usage

The tool can be run in two modes: interactive or non-interactive.

Interactive Mode

This is the easiest way to get started. If you built the binary, just run it.

cl-webnovel-to-org

If you are running from a REPL:

(ql:quickload :cl-webnovel-to-org)
(cl-webnovel-to-org:setup)

The program will then prompt you for all the necessary information:

  • The URL of the first chapter.
  • The CSS selector for the next chapter button/link.
  • The number of chapters to scrape.
  • The CSS selector for the chapter title.
  • The CSS selector for the parent of the comments section (to exclude it).
  • The title and author of the novel.

How to get CSS Selectors?

  1. Open the webnovel chapter in your browser.
  2. Right-click on the element you need (e.g., the "Next Chapter" button or a chapter title).
  3. Select Inspect or Inspect Element.
  4. In the developer tools panel, right-click on the highlighted HTML element.
  5. Go to Copy > Copy selector.
  6. Paste this value into the terminal when prompted.

Non-Interactive Mode (for Scripting)

You can provide the required information as arguments to the setup function. This is useful for scripting or if you want to save the command for later use.

(cl-webnovel-to-org:setup
  :url "https://www.example.com/novel/chapter-1"
  :button-tag "#next-chapter-link"
  :last-chap "150"
  :title-tag ".chapter-title"
  :comment-tag ".comments-area"
  :stop-after "End of Volume"
  :novel-title "My Awesome Novel"
  :novel-author "A. Writer")

Important Notes

  • Cloudflare: This tool will likely fail on websites protected by Cloudflare's anti-bot measures.
  • Recommended Sites: For a list of sites that are more likely to be compatible, you can check resources like the FMHY reading piracy guide.

License

This project is licensed under the MIT License. See the LICENSE file for details.