📚 javadocs-scraper

A TypeScript library to scrape Java objects information from a Javadocs website.

Specifically, it scrapes data (name, description, url, etc) about, and links together:

• Packages
• Classes
• Interfaces
• Object Type Parameters (Object Generics), on classes and interfaces
• Enums
  • Enum Constants
• Annotations
  • Annotation Elements
• Fields
• Methods
  • Return Type
  • Parameters
  • Type Parameters (Method Generics)

Some extra data is also calculated post scraping, like method and field inheritance.

Contents

• 📦 Installation and Usage
• 🔒 Warnings
• 🔍 Specifics

📦 Installation and Usage

1. Install with your preferred package manager:

npm install javadocs-scraper
yarn add javadocs-scraper
pnpm add javadocs-scraper
Copy

2. Instantiate a Scraper:

import { Scraper } from 'javadocs-scraper';

// From an online URL:
const urlScraper = Scraper.fromURL('https://...');

// From a local path:
const pathScraper = Scraper.fromPath('./path/to/javadocs/index.html');
Copy

import type { Fetcher } from 'javadocs-scraper';

class MyFetcher implements Fetcher {
  /** ... */
}

const myFetcher = new MyFetcher('https://...');
const scraper = Scraper.with({ fetcher: myFetcher });
Copy

3. Use the Scraper to scrape and the resulting Javadocs to access the data:

const javadocs: Javadocs = await scraper.scrape();

/** for example */
const myInterface = javadocs.getInterface('org.example.Interface');
console.log(myInterface);
/**
 * {
 *   qualifiedName: 'org.example.Interface',
 *   package: { name: 'org.example', ... },
 *   url: 'https://.../Interface.html',
 *   description: { text: 'An example interface', html: '<p>An example interface</p>' },
 *   methods: Collection {...}
 *   fields: Collection {...},
 *   typeParameters: Collection {...},
 *   // and more data, check the docs!
 * }
 */
Copy

🔒 Warnings

• Make sure to not spam a Javadocs website. It's your responsibility to not abuse the library, and implement appropiate methods to avoid abuse, like a cache.
• The scrape() method will take a while to scrape the entire website. Make sure to only run it when necessary, ideally only once in the entire program's lifecycle.

🔍 Specifics

There are distinct types of objects that hold the library together:

• A Fetcher¹, which makes requests to the Javadocs website.
• Entities², which represent a scraped object.
• QueryStrategies¹, which query the website through cheerio. Needed since HTML class and ids change between Javadoc versions.
• Scrapers¹, which scrape information from a given URL or cheerio object, to a partial object.
• Partials², which represent a partially scraped object, that is, an object without circular references to other objects.
• A ScraperCache, which caches partial objects in memory.
• Patchers¹, which patch partials to make them full entities, by linking them together.
• Javadocs, which is the final result of the scraping process.

¹ - Replaceable via constructor injection.

² - Only a type, not available in runtime.

The scraping process ocurs in the following steps:

1. A QueryStrategy is chosen by the QueryStrategyBundleFactory.
2. The RootScraper iterates through every package in the Javadocs root.
3. For every package, it's fetched, and passed to the PackageScraper.
4. The PackageScraper iterates through every class, interface, enum and annotation in the package and passes them to the appropriate Scraper.
5. Each scraper creates a partial object, and caches it in the ScraperCache.
6. Once everything is done, the Scraper uses the Patchers to patch the partial objects together, by passing the cache to each patcher.
7. The Scraper returns the patched objects, in a Javadocs object.

import { OnlineFetcher } from 'javadocs-scraper';

const scraper = Scraper.with({
  fetcher: new OnlineFetcher('https://...'), // or any other Fetcher implementation
  strategyBundleFactory: ($root: CheerioAPI) => { /** ... */ },
});
Copy

Query Strategies

Query strategies help fetch data across Java versions, without needing to write lengthy conditional code. These strategies don't actually know the Java version at runtime they're running on, and are made to support multiple at once.

In particular, the library provides two strategy "types", which are free to be extended:

Legacy strategies

For Javadocs 8 to 15. Some of the queries reassemble those of the modern strategy because of 13-15 Javadocs, which are a mix of legacy and modern, but from testing they mostly match legacy.

Legacy Javadocs don't have a consistent structure, so this strategy has a couple of workarounds, hacks and pre-compiled regexes to extract the data correctly.

Modern strategies

For Javadocs 16 to last supported (21 at the time of writing). Some of the queries reassemble those of the legacy strategy because of 16 Javadocs, which are a mix of legacy and modern, but from testing they mostly match modern.

Modern Javadocs have a more consistent structure, with classes and ids easy to query directly.