Fix SDK to working state (port gaps vs sdk-ts): validation, batch, target_meta, packaging, CI - #4
Merged
Merged
Conversation
- scrape() and scrape_async() now accept ScrapeRequest | Mapping[str, Any] - Added _to_payload() helper: model_dump(by_alias=True) for Pydantic models, dict() for mappings - Added _validate() method using jsonschema, raises decodo.errors.ValidationError on failure - scrape_batch() accepts BatchRequest | Mapping[str, Any] but skips validation (mirrors TS) - Constructor default changed to BundledSchema.shared (mirrors TS) - Removed now-unnecessary type: ignore[arg-type] comments in tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- generate_targets.py: removed generated TargetMeta pydantic class; target_meta is now emitted as dict[str, dict[str, Any]] (plain dict literals, matching TS behavior) - For each target, generate a *BatchParams class identical to *Params except url/query are typed list[str] | None instead of str | None - BatchRequest is now a proper discriminated union of all *BatchParams classes (was wrongly aliased to ScrapeRequest) - Regenerated src/decodo/generated/targets.py and request_schemas.py from IR Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Removed TargetTargetParams from __all__ (never existed, caused ImportError) - Added all 52 *BatchParams classes and BatchRequest to imports and __all__ - GoogleSearchBatchParams and BatchRequest now publicly accessible for examples Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e script - src/decodo/py.typed: empty marker file so mypy and PEP 561 consumers find types - pyproject.toml: added jsonschema>=4.0 to runtime dependencies (scrape() uses it) - pyproject.toml: added types-jsonschema>=4.0 to dev deps for mypy - pyproject.toml: added [tool.setuptools.package-data] to ship py.typed in wheel - pyproject.toml: removed [project.scripts] decodo-codegen entry (footgun: runs full network fetch + file overwrite on any invocation including --help; maintainers run via python -m decodo.codegen.codegen instead) - pyproject.toml: added ^build/ to mypy exclude to avoid duplicate-module error Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- .github/workflows/test.yml: new workflow running pytest on push+PR, Python 3.12, installs .[dev]; mirrors lint.yml style - .github/workflows/worklfow.yml: added contents: read permission so actions/checkout can access the private repo (id-token: write alone zeros contents, causing 'Repository not found' on checkout) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
README.md: - All DecodoClient(...) calls now use DecodoConfig(web_scraping_api=WebScrapingApiConfig(token=...)) - Quick start and all API snippets use typed Params objects (GoogleSearchParams etc.) - Batch snippet uses GoogleSearchBatchParams with query as list - Error handling snippet fixed to use typed params - Documented that token is base64-encoded user:password from dashboard - Added git install fallback (pip install from GitHub) until PyPI publish - Removed git conflict marker line (>>>>>>> 03b68da) examples/web_scraping_api/batch/google_search_batch.py: - Import GoogleSearchBatchParams instead of GoogleSearchParams - Removed duplicate print('Polling for results...') before the loop Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JuliusTva
force-pushed
the
fix/working-sdk-port-gaps
branch
6 times, most recently
from
July 27, 2026 10:57
04ee7bc to
f33d39e
Compare
JuliusTva
force-pushed
the
fix/working-sdk-port-gaps
branch
from
July 27, 2026 11:00
f33d39e to
8bb8e4a
Compare
JuliusTva
force-pushed
the
fix/working-sdk-port-gaps
branch
from
July 30, 2026 12:30
c8ab557 to
e4b4da4
Compare
JuliusTva
force-pushed
the
fix/working-sdk-port-gaps
branch
2 times, most recently
from
July 31, 2026 07:21
9b31d3a to
c7cb001
Compare
JuliusTva
force-pushed
the
fix/working-sdk-port-gaps
branch
from
July 31, 2026 07:23
c7cb001 to
0a188f7
Compare
ASDCD154
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the Python SDK actually work and brings it in line with the reference
sdk-ts. Julius ported the TS SDK but a few things were left incomplete/broken: 4 unit tests failing onmain, batch unusable, package not type-marked, CI not running tests, and the publish workflow misconfigured. All fixes below were cross-checked againstDecodo/sdk-tsas the ground truth.Fixes
Correctness (were failing tests / broken features)
scrape/scrape_async. Ported fromsdk-tsweb-scraping-api.tsvalidate()— the Python port had storedself._schemabut never used it. Now accepts a typed*Paramsmodel or a plain dict, validates against the target's JSON schema viajsonschema, raisesValidationErrorbefore the HTTP call.scrape_batchdoes not validate (matches TS). Fixes 3 tests.target_metaemitted as plain dicts, not a Pydantic model. Codegen was generating aTargetMeta(BaseModel)that collided with theTargetMetaTypedDict inschema/types.pyand brokemeta["group"]subscripting. TS emits plain objects — codegen now matches. Fixes 1 test.BatchRequest. Was aliasedBatchRequest = ScrapeRequest, soquery/urlstayedstrand batch rejected list input client-side. Codegen now generates per-target*BatchParams(query/url aslist[str]) and a proper discriminatedBatchRequest, mirroring the TS mapped type. The bundled batch example now runs.__init__.__all__: removed phantomTargetTargetParams(ImportError), exported the 52*BatchParams+BatchRequest.Packaging / release
py.typedmarker added + shipped viapackage-data(SDK was untyped to consumers despite full annotations).jsonschema>=4.0promoted to a runtime dependency (validation needs it);types-jsonschemaadded to dev.contents: readtoworklfow.yml. This is the confirmed root cause of the failed PyPI publish — on a private repo, declaring onlyid-token: writezeroescontents, soactions/checkoutfails with "Repository not found". (Filename typo left as-is to avoid breaking the future Trusted Publisher binding.)decodo-codegenconsole script — it had no argparse and would silently run a full codegen (network + overwrite generated files, into site-packages when installed) on any invocation incl.--help. Codegen stays runnable viapython -m decodo.codegen.codegenfor maintainers.Tooling / docs
pytest(test.yml) on push + PR — the gap that let the failing tests ship unnoticed.DecodoClient(web_scraping_api={...})→DecodoConfig(...)), scrape snippets now use typed params, removed a stray git merge-conflict marker, documented thattokenis the base64user:passbasic-auth token, added a git-install fallback while the package isn't on PyPI.Verification
pytest: 21 passed (was 4 failed / 17 passed)ruff check: clean ·mypy(strict, repo config): cleanpy.typedpresent in artifactGoogleSearchBatchParams(query=[...])accepted;GoogleSearchParams(query=[...])still correctly rejected__all__entry importableDeliberately NOT changed (flagging for a call, not silently altering)
TimeoutErrordoes not inheritDecodoError, andAuthenticationError/ValidationErrorhardcode 401/422 even for 403/400 responses. Both are faithful tosdk-tsso I left them, but they're real usability sharp edges in both SDKs — worth deciding cross-SDK (aTimeoutErrorthat escapesexcept DecodoErrorwill surprise users).requires-python >=3.12: no 3.12-only syntax in the code; deps allow older. Could relax to>=3.10to widen adoption — left as-is pending a support-policy decision.Still needed outside this PR
contents: readfix unblocks checkout, but the project also doesn't exist on PyPI yet. This is a PyPI-side action, not a code change.🤖 Generated with Claude Code