Compare commits

...
19 changed files with 457 additions and 51 deletions
+3
View File
@@ -24,6 +24,9 @@ jobs:
- name: Cache cargo
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- name: Check package version sync
run: python3 scripts/version.py --check
- name: Run tests
run: cargo test --verbose
+3
View File
@@ -31,6 +31,9 @@ jobs:
with:
fetch-depth: 2
- name: Check package version sync
run: python3 scripts/version.py --check
- name: Check if version changed
id: check
run: |
+3
View File
@@ -28,6 +28,9 @@ jobs:
with:
fetch-depth: 2
- name: Check package version sync
run: python3 scripts/version.py --check
- name: Check if version changed
id: check
run: |
+3
View File
@@ -28,6 +28,9 @@ jobs:
with:
fetch-depth: 2
- name: Check package version sync
run: python3 scripts/version.py --check
- name: Check package version
id: check
run: |
+3
View File
@@ -28,6 +28,9 @@ jobs:
with:
fetch-depth: 2
- name: Check package version sync
run: python3 scripts/version.py --check
- name: Check if version changed
id: check
run: |
+1 -2
View File
@@ -31,9 +31,8 @@ Thumbs.db
napi/index.js
napi/index.d.ts
# Local samples and scripts
# Local samples
samples/
scripts/
# Test output
test_output/
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector"
version = "0.1.8"
version = "1.14.0"
edition = "2021"
autobins = false
authors = ["Firecrawl Team"]
+1 -1
View File
@@ -110,7 +110,7 @@ Or add it manually:
```toml
[dependencies]
pdf-inspector = "0.1"
pdf-inspector = "1"
```
```rust
+43 -24
View File
@@ -1,38 +1,57 @@
# Publishing
The Rust crate is published to [crates.io](https://crates.io/crates/pdf-inspector) with trusted publishing from GitHub Actions. The first release was published manually; future releases publish from `.github/workflows/publish-crate.yml` when a `Cargo.toml` version change lands on `main`.
Every pdf-inspector distribution uses one shared semantic version:
## crates.io Trusted Publisher
- Rust crate: `pdf-inspector`
- Python package: `pdf-inspector`
- Node package: `@firecrawl/pdf-inspector` and its platform packages
- Browser package: `@firecrawl/pdf-inspector-wasm`
- Internal NAPI and WASM Rust crates
Configure the trusted publisher for the `pdf-inspector` crate with:
`Cargo.toml` is the canonical version source. Update every manifest and lockfile
with:
- Repository: `firecrawl/pdf-inspector`
- Workflow: `publish-crate.yml`
- Environment: `crates-io`
```bash
python3 scripts/version.py <version>
```
The workflow uses `rust-lang/crates-io-auth-action@v1` to exchange GitHub's OIDC token for a short-lived crates.io token, then passes it to `cargo publish`.
Verify that nothing has diverged with:
## Release Steps
```bash
python3 scripts/version.py --check
```
1. Update `version` in `Cargo.toml`.
2. Merge the version bump to `main`.
3. The publish workflow compares the new `Cargo.toml` version with `HEAD~1`, runs `cargo publish --dry-run`, then publishes if that version is not already on crates.io.
CI and every publishing workflow run this check before building or publishing.
If `Cargo.toml` changes without a package version bump, the workflow exits without publishing.
## Release steps
## Browser WebAssembly package
1. Choose the next shared semantic version and run `scripts/version.py`.
2. Review the manifest and lockfile changes in the version-bump pull request.
3. Merge the pull request to `main`.
4. The crates.io, PyPI, Node, and WASM workflows independently build and
publish that version from the same commit.
5. After all registries succeed, create one `v<version>` GitHub release that
links to each package and describes changes since the previous shared tag.
The browser package is published as `@firecrawl/pdf-inspector-wasm`. Its version lives in `wasm/Cargo.toml`, and `.github/workflows/publish-wasm.yml` builds the `web` target with `wasm-pack` before publishing the generated package.
The independent workflows are intentionally idempotent. A manual dispatch from
`main` can repair a partial release, and already-published artifacts are skipped.
The npm package must exist before a trusted publisher can be configured. For the first release only:
## Trusted publishers
1. Build with `wasm-pack build wasm --target web --scope firecrawl --out-dir pkg --release`.
2. Inspect with `npm pack --dry-run ./wasm/pkg`.
3. Publish with `npm publish ./wasm/pkg --access public` from an authorized maintainer session.
4. In the package settings on npm, configure the GitHub Actions trusted publisher:
- Organization: `firecrawl`
- Repository: `pdf-inspector`
- Workflow: `publish-wasm.yml`
- Allowed action: `npm publish`
The repositories use GitHub Actions OIDC instead of long-lived registry tokens.
Configure each registry's trusted publisher for `firecrawl/pdf-inspector` and
its corresponding workflow:
After that one-time bootstrap, bumping the version in `wasm/Cargo.toml` and merging it to `main` publishes through OIDC. Until the package exists, the workflow exits cleanly without attempting an unauthenticated first publish. See npm's [trusted publishing documentation](https://docs.npmjs.com/trusted-publishers/) for the registry-side setup.
- crates.io: `publish-crate.yml`, environment `crates-io`
- PyPI: `publish-pypi.yml`, environment `pypi`
- npm Node package: `publish.yml`
- npm WASM package: `publish-wasm.yml`
The WASM package must exist before npm trusted publishing can be configured. If
it ever needs to be bootstrapped again, build and inspect it before publishing:
```bash
wasm-pack build wasm --target web --scope firecrawl --out-dir pkg --release
npm pack --dry-run ./wasm/pkg
npm publish ./wasm/pkg --access public
```
+2 -2
View File
@@ -851,7 +851,7 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "pdf-inspector"
version = "0.1.8"
version = "1.14.0"
dependencies = [
"env_logger",
"include_dir",
@@ -867,7 +867,7 @@ dependencies = [
[[package]]
name = "pdf-inspector-napi"
version = "0.2.3"
version = "1.14.0"
dependencies = [
"napi",
"napi-build",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector-napi"
version = "0.2.3"
version = "1.14.0"
edition = "2021"
[lib]
+6 -6
View File
@@ -8,12 +8,12 @@
"@napi-rs/cli": "^3.4.1",
},
"optionalDependencies": {
"@firecrawl/pdf-inspector-darwin-arm64": "1.13.0",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.13.0",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.13.0",
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.13.0",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.13.0",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.13.0",
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.0",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.0",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.0",
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.0",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.0",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.0",
},
},
},
+7 -7
View File
@@ -1,6 +1,6 @@
{
"name": "@firecrawl/pdf-inspector",
"version": "1.13.0",
"version": "1.14.0",
"description": "Fast PDF classification and text extraction. Detect text-based vs scanned PDFs, extract text by region with quality checks. Native Rust performance via napi-rs.",
"main": "index.js",
"types": "index.d.ts",
@@ -52,11 +52,11 @@
"@napi-rs/cli": "^3.4.1"
},
"optionalDependencies": {
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.13.0",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.13.0",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.13.0",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.13.0",
"@firecrawl/pdf-inspector-darwin-arm64": "1.13.0",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.13.0"
"@firecrawl/pdf-inspector-linux-x64-gnu": "1.14.0",
"@firecrawl/pdf-inspector-linux-x64-musl": "1.14.0",
"@firecrawl/pdf-inspector-linux-arm64-gnu": "1.14.0",
"@firecrawl/pdf-inspector-linux-arm64-musl": "1.14.0",
"@firecrawl/pdf-inspector-darwin-arm64": "1.14.0",
"@firecrawl/pdf-inspector-win32-x64-msvc": "1.14.0"
}
}
+3 -3
View File
@@ -4,9 +4,9 @@ build-backend = "maturin"
[project]
name = "pdf-inspector"
# Bump this to publish to PyPI — CI publishes automatically when the version
# changes on main (same flow as napi/package.json for npm).
version = "0.2.7"
# Keep package versions in sync with `python3 scripts/version.py <version>`.
# CI publishes automatically when the synchronized change lands on main.
version = "1.14.0"
description = "Fast PDF inspection, classification, and text extraction with smart scanned vs text-based detection"
readme = "docs/python.md"
license = { text = "MIT" }
+111
View File
@@ -0,0 +1,111 @@
import json
import sys
import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from version import PLATFORM_PACKAGES, check_versions, set_versions
class VersionTests(unittest.TestCase):
def setUp(self):
self.temporary = tempfile.TemporaryDirectory()
self.root = Path(self.temporary.name)
(self.root / "napi").mkdir()
(self.root / "site").mkdir()
(self.root / "wasm").mkdir()
self._write_manifest("Cargo.toml", "package", "0.1.0")
self._write_manifest("pyproject.toml", "project", "0.1.0")
self._write_manifest("napi/Cargo.toml", "package", "0.1.0")
self._write_manifest("wasm/Cargo.toml", "package", "0.1.0")
package = {
"name": "@firecrawl/pdf-inspector",
"version": "0.1.0",
"optionalDependencies": {
dependency: "0.1.0" for dependency in PLATFORM_PACKAGES
},
}
(self.root / "napi/package.json").write_text(
json.dumps(package), encoding="utf-8"
)
(self.root / "napi/bun.lock").write_text(
"\n".join(
f' "{dependency}": "0.1.0",'
for dependency in PLATFORM_PACKAGES
)
+ "\n",
encoding="utf-8",
)
(self.root / "site/index.html").write_text(
'https://cdn.jsdelivr.net/npm/@firecrawl/pdf-inspector-wasm@0.1.0/'
'pdf_inspector_wasm.js\n',
encoding="utf-8",
)
self._write_lock(
"napi/Cargo.lock", ("pdf-inspector", "pdf-inspector-napi")
)
self._write_lock(
"wasm/Cargo.lock", ("pdf-inspector", "pdf-inspector-wasm")
)
def tearDown(self):
self.temporary.cleanup()
def _write_manifest(self, relative, section, version):
(self.root / relative).write_text(
f'[{section}]\nname = "fixture"\nversion = "{version}"\n',
encoding="utf-8",
)
def _write_lock(self, relative, packages):
content = "\n".join(
f'[[package]]\nname = "{package}"\nversion = "0.1.0"\n'
for package in packages
)
(self.root / relative).write_text(content, encoding="utf-8")
def test_updates_every_version_location(self):
set_versions("1.14.0", self.root)
self.assertEqual(check_versions(self.root), "1.14.0")
def test_reports_a_divergent_package(self):
self._write_manifest("wasm/Cargo.toml", "package", "0.2.0")
with self.assertRaisesRegex(ValueError, "WASM package: 0.2.0"):
check_versions(self.root)
def test_rejects_an_invalid_version(self):
with self.assertRaisesRegex(ValueError, "Invalid semantic version"):
set_versions("next", self.root)
def test_rejects_numeric_prerelease_with_leading_zero(self):
before = (self.root / "Cargo.toml").read_text(encoding="utf-8")
with self.assertRaisesRegex(ValueError, "Invalid semantic version"):
set_versions("1.2.3-01", self.root)
self.assertEqual(
(self.root / "Cargo.toml").read_text(encoding="utf-8"), before
)
def test_preflight_failure_does_not_partially_update(self):
before = (self.root / "Cargo.toml").read_text(encoding="utf-8")
(self.root / "site/index.html").write_text(
"missing module URL\n", encoding="utf-8"
)
with self.assertRaisesRegex(ValueError, "Missing pinned WASM package URL"):
set_versions("1.14.0", self.root)
self.assertEqual(
(self.root / "Cargo.toml").read_text(encoding="utf-8"), before
)
if __name__ == "__main__":
unittest.main()
+262
View File
@@ -0,0 +1,262 @@
#!/usr/bin/env python3
"""Keep every pdf-inspector package on one release version."""
from __future__ import annotations
import argparse
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
PRERELEASE_IDENTIFIER = (
r"(?:0|[1-9]\d*|[0-9A-Za-z-]*[A-Za-z-][0-9A-Za-z-]*)"
)
SEMVER = re.compile(
r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)"
rf"(?:-{PRERELEASE_IDENTIFIER}(?:\.{PRERELEASE_IDENTIFIER})*)?"
r"(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$"
)
VERSION_LINE = re.compile(r'^(\s*version\s*=\s*")[^"]+(".*)$')
SECTION_LINE = re.compile(r"^\s*\[([^]]+)]\s*$")
PLATFORM_PACKAGES = (
"@firecrawl/pdf-inspector-linux-x64-gnu",
"@firecrawl/pdf-inspector-linux-x64-musl",
"@firecrawl/pdf-inspector-linux-arm64-gnu",
"@firecrawl/pdf-inspector-linux-arm64-musl",
"@firecrawl/pdf-inspector-darwin-arm64",
"@firecrawl/pdf-inspector-win32-x64-msvc",
)
TOML_VERSIONS = (
("Rust crate", Path("Cargo.toml"), "package"),
("Python package", Path("pyproject.toml"), "project"),
("NAPI crate", Path("napi/Cargo.toml"), "package"),
("WASM package", Path("wasm/Cargo.toml"), "package"),
)
LOCK_VERSIONS = (
("NAPI lock: core", Path("napi/Cargo.lock"), "pdf-inspector"),
("NAPI lock: binding", Path("napi/Cargo.lock"), "pdf-inspector-napi"),
("WASM lock: core", Path("wasm/Cargo.lock"), "pdf-inspector"),
("WASM lock: binding", Path("wasm/Cargo.lock"), "pdf-inspector-wasm"),
)
SITE_WASM_VERSION = re.compile(
r"(@firecrawl/pdf-inspector-wasm@)([^/\"]+)(/pdf_inspector_wasm\.js)"
)
def _read_section_version(path: Path, section: str) -> str:
active = False
for line in path.read_text(encoding="utf-8").splitlines():
section_match = SECTION_LINE.match(line)
if section_match:
active = section_match.group(1) == section
elif active:
version_match = VERSION_LINE.match(line)
if version_match:
return line.split('"', 2)[1]
raise ValueError(f"No version found in [{section}] of {path}")
def _write_section_version(path: Path, section: str, version: str) -> None:
lines = path.read_text(encoding="utf-8").splitlines(keepends=True)
active = False
for index, line in enumerate(lines):
section_match = SECTION_LINE.match(line)
if section_match:
active = section_match.group(1) == section
elif active:
version_match = VERSION_LINE.match(line)
if version_match:
newline = "\n" if line.endswith("\n") else ""
replacement = (
f"{version_match.group(1)}{version}"
f"{version_match.group(2).rstrip()}"
)
lines[index] = (
f"{replacement}{newline}"
)
path.write_text("".join(lines), encoding="utf-8")
return
raise ValueError(f"No version found in [{section}] of {path}")
def _package_block(lines: list[str], package: str) -> tuple[int, int]:
for start, line in enumerate(lines):
if line.strip() != "[[package]]":
continue
end = next(
(
index
for index in range(start + 1, len(lines))
if lines[index].strip() == "[[package]]"
),
len(lines),
)
if any(line.strip() == f'name = "{package}"' for line in lines[start:end]):
return start, end
raise ValueError(f"No lockfile entry found for {package}")
def _read_lock_version(path: Path, package: str) -> str:
lines = path.read_text(encoding="utf-8").splitlines()
start, end = _package_block(lines, package)
for line in lines[start:end]:
version_match = VERSION_LINE.match(line)
if version_match:
return line.split('"', 2)[1]
raise ValueError(f"No version found for {package} in {path}")
def _write_lock_version(path: Path, package: str, version: str) -> None:
lines = path.read_text(encoding="utf-8").splitlines(keepends=True)
start, end = _package_block(lines, package)
for index in range(start, end):
version_match = VERSION_LINE.match(lines[index])
if version_match:
newline = "\n" if lines[index].endswith("\n") else ""
lines[index] = (
f'{version_match.group(1)}{version}{version_match.group(2).rstrip()}'
f"{newline}"
)
path.write_text("".join(lines), encoding="utf-8")
return
raise ValueError(f"No version found for {package} in {path}")
def _node_versions(root: Path) -> dict[str, str]:
package = json.loads((root / "napi/package.json").read_text(encoding="utf-8"))
versions = {"Node package": package["version"]}
optional = package.get("optionalDependencies", {})
for dependency in PLATFORM_PACKAGES:
if dependency not in optional:
raise ValueError(f"Missing Node optional dependency: {dependency}")
versions[f"Node optional dependency: {dependency}"] = optional[dependency]
return versions
def _bun_versions(root: Path) -> dict[str, str]:
text = (root / "napi/bun.lock").read_text(encoding="utf-8")
versions = {}
for dependency in PLATFORM_PACKAGES:
match = re.search(
rf'"{re.escape(dependency)}": "([^"]+)"[,]', text
)
if not match:
raise ValueError(f"Missing Bun lock dependency: {dependency}")
versions[f"Bun lock: {dependency}"] = match.group(1)
return versions
def _site_wasm_version(root: Path) -> str:
text = (root / "site/index.html").read_text(encoding="utf-8")
match = SITE_WASM_VERSION.search(text)
if not match:
raise ValueError("Missing pinned WASM package URL in site/index.html")
return match.group(2)
def package_versions(root: Path = ROOT) -> dict[str, str]:
versions = {
label: _read_section_version(root / relative, section)
for label, relative, section in TOML_VERSIONS
}
versions.update(_node_versions(root))
versions.update(_bun_versions(root))
versions["Website WASM module"] = _site_wasm_version(root)
versions.update(
{
label: _read_lock_version(root / relative, package)
for label, relative, package in LOCK_VERSIONS
}
)
return versions
def check_versions(root: Path = ROOT) -> str:
versions = package_versions(root)
expected = versions["Rust crate"]
if not SEMVER.fullmatch(expected):
raise ValueError(f"Rust crate has an invalid semantic version: {expected}")
mismatches = {
label: version for label, version in versions.items() if version != expected
}
if mismatches:
details = "\n".join(
f" - {label}: {version}" for label, version in mismatches.items()
)
raise ValueError(f"Expected every package to use {expected}:\n{details}")
return expected
def set_versions(version: str, root: Path = ROOT) -> None:
if not SEMVER.fullmatch(version):
raise ValueError(f"Invalid semantic version: {version}")
# Validate every expected location before writing the first file. This
# prevents a stale manifest or generated file from leaving a partial bump.
package_versions(root)
for _, relative, section in TOML_VERSIONS:
_write_section_version(root / relative, section, version)
package_path = root / "napi/package.json"
package = json.loads(package_path.read_text(encoding="utf-8"))
package["version"] = version
optional = package.get("optionalDependencies", {})
for dependency in PLATFORM_PACKAGES:
if dependency not in optional:
raise ValueError(f"Missing Node optional dependency: {dependency}")
optional[dependency] = version
package_path.write_text(json.dumps(package, indent=2) + "\n", encoding="utf-8")
bun_path = root / "napi/bun.lock"
bun_text = bun_path.read_text(encoding="utf-8")
for dependency in PLATFORM_PACKAGES:
pattern = rf'("{re.escape(dependency)}": ")[^"]+("[,])'
bun_text, count = re.subn(
pattern, rf"\g<1>{version}\g<2>", bun_text, count=1
)
if count != 1:
raise ValueError(f"Missing Bun lock dependency: {dependency}")
bun_path.write_text(bun_text, encoding="utf-8")
site_path = root / "site/index.html"
site_text = site_path.read_text(encoding="utf-8")
site_text, count = SITE_WASM_VERSION.subn(
rf"\g<1>{version}\g<3>", site_text, count=1
)
if count != 1:
raise ValueError("Missing pinned WASM package URL in site/index.html")
site_path.write_text(site_text, encoding="utf-8")
for _, relative, package_name in LOCK_VERSIONS:
_write_lock_version(root / relative, package_name, version)
check_versions(root)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("version", nargs="?", help="new shared semantic version")
parser.add_argument(
"--check", action="store_true", help="fail if package versions have diverged"
)
arguments = parser.parse_args()
if arguments.check == bool(arguments.version):
parser.error("provide either a version or --check")
try:
if arguments.check:
version = check_versions()
print(f"All packages use {version}")
else:
set_versions(arguments.version)
print(f"Updated all packages to {arguments.version}")
except ValueError as error:
parser.exit(1, f"{error}\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+1 -1
View File
@@ -975,7 +975,7 @@ result = pdf_inspector.<span class="fn">process_pdf</span>(<span class="str">"do
<script>
(() => {
const MAX_FILE_SIZE = 25 * 1024 * 1024;
const WASM_MODULE_URL = "https://cdn.jsdelivr.net/npm/@firecrawl/pdf-inspector-wasm@0.1.1/pdf_inspector_wasm.js";
const WASM_MODULE_URL = "https://cdn.jsdelivr.net/npm/@firecrawl/pdf-inspector-wasm@1.14.0/pdf_inspector_wasm.js";
const input = document.querySelector("#pdf-input");
const dropZone = document.querySelector("#drop-zone");
const filePanel = document.querySelector("#demo-file");
+2 -2
View File
@@ -724,7 +724,7 @@ checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
[[package]]
name = "pdf-inspector"
version = "0.1.8"
version = "1.14.0"
dependencies = [
"env_logger",
"include_dir",
@@ -740,7 +740,7 @@ dependencies = [
[[package]]
name = "pdf-inspector-wasm"
version = "0.1.4"
version = "1.14.0"
dependencies = [
"console_error_panic_hook",
"js-sys",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "pdf-inspector-wasm"
version = "0.1.4"
version = "1.14.0"
edition = "2021"
authors = ["Firecrawl Team"]
description = "Browser WebAssembly bindings for pdf-inspector"