feat: add Telegraph site support - #147
Conversation
| page = envelope.get("result") | ||
| if not isinstance(page, dict): | ||
| raise NazurinError("Invalid Telegraph API response") | ||
| if not isinstance(page.get("path"), str): | ||
| raise NazurinError("Invalid Telegraph API response") | ||
| if not isinstance(page.get("title"), str) or not page["title"]: | ||
| raise NazurinError("Invalid Telegraph API response") | ||
| if not isinstance(page.get("content"), list): | ||
| raise NazurinError("Invalid Telegraph API response") |
There was a problem hiding this comment.
Consider using Pydantic model to validate response.
| if ( | ||
| parsed.scheme.lower() not in {"http", "https"} | ||
| or host not in SUPPORTED_HOSTS | ||
| or parsed.username is not None | ||
| or parsed.password is not None | ||
| or port is not None | ||
| ): | ||
| raise NazurinError("Invalid Telegraph URL") |
There was a problem hiding this comment.
This seems unnecessary, invalid URL won't match regex pattern in the first place.
| SUPPORTED_HOSTS = {"graph.org", "telegra.ph"} | ||
|
|
||
| patterns = [ | ||
| r"(?P<url>(?i:https?://(?:telegra\.ph|graph\.org)/[^\s,]+))", |
There was a problem hiding this comment.
I feel we should add more constraints on the page path so we can eliminate some validation logic.
Based on my rough test, these characters will be removed when creating path from title:
./\?=&()[]{}:'"
So it seems [\w%-]+ is enough, \w will match unicode word characters, and % will match quoted URLs, then validate_page_path can be simplified.
| @network_retry | ||
| async def get_page(self, page_path: str) -> tuple[dict, dict]: | ||
| """Fetch a Telegraph page and return the raw envelope and page data.""" | ||
| api_url = f"{self.API_BASE}/getPage/{quote(page_path, safe='')}" |
There was a problem hiding this comment.
I believe aiohttp can handle unquoted URL, so we would prefer that everywhere.
| "path_hash": hashlib.sha256(page["path"].encode()).hexdigest()[ | ||
| :PATH_HASH_LENGTH | ||
| ], |
There was a problem hiding this comment.
Should make a helper function for this, also duplicated in build_archive_name.
| source_url: str, | ||
| destination: str, | ||
| ): | ||
| page_url = safe_url(page.get("url"), source_url, SAFE_MEDIA_SCHEMES) |
There was a problem hiding this comment.
I don't think we need to validate the URL returned by Telegraph, it should always be the canonical form.
| workspace = Path(self._workspace) if self._workspace else None | ||
| if workspace and await aiofiles.os.path.exists(workspace): | ||
| await async_wrap(shutil.rmtree)(workspace, ignore_errors=True) | ||
| for directory in (workspace.parent, workspace.parent.parent): |
There was a problem hiding this comment.
This seems error-prone, we should just recursively remove a single workspace temporary folder, if we don't put it inside {archive_name} we don't need to remove the parent.
| if tag in DIRECT_TAGS: | ||
| return f"<{tag}>{rendered_children}</{tag}>" | ||
| if tag in VOID_TAGS: | ||
| return f"<{tag}>" |
There was a problem hiding this comment.
| return f"<{tag}>" | |
| return f"<{tag}/>" |
| return rendered_children | ||
|
|
||
| tag = TAG_ALIASES.get(tag, tag) | ||
| if tag in DIRECT_TAGS: |
| rendered_children = ( | ||
| self._render_nodes(children) if isinstance(children, list) else "" | ||
| ) | ||
| if not isinstance(tag, str): |
There was a problem hiding this comment.
When will tag not be a string? Document only mentions string type.
2 participants
Add Telegraph support (
telegra.phandgraph.org). This site plugin fetches Telegraph pages through the official Telegraph API, saves Telegraph page data (title, original URL, etc.) in JSON, downloads the original page images (mostly WebP), and then renders each page as an HTML file.Changes:
nazurin/sites/telegraph/, including:interface.pyfor matching and normalising Telegraph URLs.api.pyfor retrieving page data through the official Telegraph API.models.pyfor preparing page archives and downloading images.renderer.pyfor converting Telegraph content into HTML.config.pyand__init__.pyfor site configuration and plugin registration.nazurin/models/file.pyto support an optional custom local path, allowing each Telegraph page to use its own temporary workspace without changing the default behaviour of existing sites..env.exampleand other doc files to add Telegraph configuration and usage information.Tested with multiple Telegraph links using Docker, with both Local and Mega storage backends and the MongoDB.