📚 javadocs-scraper

A TypeScript library to scrape Java objects information from a Javadocs website.

Specifically, it scrapes data (name, description, url, etc) about, and links together:

• Packages
• Classes
• Interfaces
• Object Type Parameters (Object Generics), on classes and interfaces
• Enums
  • Enum Constants
• Annotations
  • Annotation Elements
• Fields
• Methods
  • Return Type
  • Parameters
  • Type Parameters (Method Generics)

Some extra data is also calculated post scraping, like method and field inheritance.

Contents

• 📦 Installation and Usage
• 🔒 Warnings
• 🔍 Specifics

📦 Installation and Usage

1. Install with your preferred package manager:

npm install javadocs-scraper
yarn add javadocs-scraper
pnpm add javadocs-scraper
Copy

2. Instantiate a Scraper:

import { Scraper } from 'javadocs-scraper';

const scraper = Scraper.fromURL('https://...');
Copy

import type { Fetcher } from 'javadocs-scraper';

class MyFetcher implements Fetcher {
  /** ... */
}

const myFetcher = new MyFetcher('https://...');
const scraper = Scraper.with({ fetcher: myFetcher });
Copy

3. Use the Scraper to scrape information:

const javadocs: Javadocs = await scraper.scrape();

/** for example */
const myInterface = javadocs.getInterface('org.example.Interface');
Copy

🔒 Warnings

• Make sure to not spam a Javadocs website. It's your responsibility to not abuse the library, and implement appropiate methods to avoid abuse, like a cache.
• The scrape() method will take a while to scrape the entire website. Make sure to only run it when necessary, ideally only once in the entire program's lifecycle.

🔍 Specifics

There are distinct types of objects that hold the library together:

• A Fetcher¹, which makes requests to the Javadocs website.
• Entities², which represent a scraped object.
• QueryStrategies¹, which query the website through cheerio. Needed since HTML class and ids change between Javadoc versions.
• Scrapers¹, which scrape information from a given URL or cheerio object, to a partial object.
• Partials², which represent a partially scraped object, that is, an object without circular references to other objects.
• A ScraperCache, which caches partial objects in memory.
• Patchers¹, which patch partials to make them full entities, by linking them together.
• Javadocs, which is the final result of the scraping process.

¹ - Replaceable via constructor injection.

² - Only a type, not available in runtime.

The scraping process ocurs in the following steps:

1. A QueryStrategy is chosen by the QueryStrategyFactory.
2. The RootScraper iterates through every package in the Javadocs root.
3. For every package, it's fetched, and passed to the PackageScraper.
4. The PackageScraper iterates through every class, interface, enum and annotation in the package and passes them to the appropriate Scraper.
5. Each scraper creates a partial object, and caches it in the ScraperCache.
6. Once everything is done, the Scraper uses the Patchers to patch the partial objects together, by passing the cache to each patcher.
7. The Scraper returns the patched objects, in a Javadocs object.

import { OnlineFetcher } from 'javadocs-scraper';

const myFetcher = new OnlineFetcher('https://...');
const factory = new MyQueryStrategyFactory();
const scraper = Scraper.with({
  fetcher: myFetcher,
  queryStrategyFactory: factory
});
Copy