docs: document new features + bump to 1.5.0 (and fix extract on Python 3.9) - #125
Merged
Conversation
… 3.9 Documents the public features now on main and bumps the version (minor): - Selector parser (#123): CSS/XPath, find_all, find_by_text, find_similar - pyscrappy extract CLI (URL -> .md/.json/.txt/.html) - ScraperConfig(impersonate=...) TLS fingerprint impersonation (#124) README gains key-feature bullets, a Selector quick-start section, the pyscrappy[stealth] extra, a command-line extract section, the impersonate config option, and a stealth subsection under Proxies and blocked sites. Version bumped to 1.5.0 across pyproject, __init__, and server.json. Also fixes CI on Python 3.9: extract lived in mcp/agent.py, which imports the MCP stack (fastmcp, 3.10+), so importing it broke 3.9. Moved run_extract + the CLI entry to a new MCP-free pyscrappy/cli.py (entry point pyscrappy.cli:main); chat still uses the agent, imported lazily.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the public features now on
mainand bumps the version to 1.5.0 (minor). Also fixes CI on Python 3.9 for theextractcommand.README
Adds documentation for the three recent features:
Selectorparser (feat(parser): add a chainable Selector for direct HTML navigation #123): CSS/XPath selection,find_all,find_by_text,find_similar, chaining — a new quick-start section plus a key-feature bullet.pyscrappy extractCLI (feat: addpyscrappy extractCLI and TLS-fingerprint impersonation #124): scrape a URL straight to.md/.json/.txt/.html, with--css-selector/--render-js— a command-line section.pyscrappy extractCLI and TLS-fingerprint impersonation #124):ScraperConfig(impersonate="chrome")— theimpersonateconfig option, a stealth subsection under Proxies and blocked sites, and thepyscrappy[stealth]extra in Installation.Version
Bumped to 1.5.0 across
pyproject.toml,src/pyscrappy/__init__.py, andserver.json(both entries). Minor, since these are new backward-compatible public features.Fix:
extracton Python 3.9extractwas defined inmcp/agent.py, which imports the MCP stack (fastmcp, gated to Python >=3.10) at module load.extracthas no MCP dependency, but importing it there pulled in MCP, so test collection and the command failed on 3.9 (CI red on the last PR).Moved
run_extract+ the CLI entry point to a new MCP-freepyscrappy/cli.py(entry point is nowpyscrappy.cli:main). Thechatcommand still uses the MCP agent, now imported lazily only whenchatruns, with a clear hint to installpyscrappy[mcp]if absent.mcp/agent.pykeeps the full agent unchanged.Verification
Full suite: 457 passed, 19 deselected.
ruff check+ruff formatclean.run_extractconfirmed importable without loading MCP (the exact 3.9 breakage).