diff --git a/README.md b/README.md index bad495e..badbc6f 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,8 @@ - [Configuration](#configuration) - [Git configuration](#git-configuration) - [SSH configuration](#ssh-configuration) + - [GitHub OAuth configuration](#github-oauth-configuration) + - [Choose "Only select repositories"](#choose-only-select-repositories) - [Example: Running code saved in custom repository + template 🧩](#example-running-code-saved-in-custom-repository--template-) - [Example: Listing preinstalled packages](#example-listing-preinstalled-packages) - [Example: Accessing custom configuration parameters](#example-accessing-custom-configuration-parameters) @@ -79,13 +81,17 @@ configured and executed directly in Keboola. This eliminates the need to build a The git configuration object supports the following parameters: -- `url`: Repository URL – supports both HTTPS and SSH formats. +- `url`: Repository URL – supports both HTTPS and SSH formats (all `auth` methods except `oauth`). +- `repository`: Clone URL of the repository, picked from a list of what the GitHub App may read + (`"auth": "oauth"` only). Replaces `url` for OAuth configurations. - `branch`: Branch name to checkout – UI provides branch selection. - `filename`: Python script filename to execute – UI lists available files. - `auth`: Repository visibility & authentication method. - `none`: Public repository, no authentication (default). - `pat`: Private repository, Personal Access Token. - `ssh`: Private repository, SSH key. + - `oauth`: Private GitHub repository authorized via the **Authorization** section of the configuration. + Requires the Keboola GitHub App to be installed first – see [GitHub OAuth configuration](#github-oauth-configuration). - `#token`: Personal Access Token (`"auth": "pat"` only). This value will be encrypted in Keboola Storage. The same token also authenticates private git dependencies declared in `[tool.uv.sources]` in your `pyproject.toml`, so there is no need to embed tokens directly in the source file. @@ -99,6 +105,58 @@ The git configuration object supports the following parameters: - `#private`: Private key used for authentication. This value will be encrypted in Keboola Storage. +### GitHub OAuth configuration + +With `"auth": "oauth"` no credential is entered into the configuration at all – the access token is issued by +the Keboola OAuth broker. Only `https://github.com` URLs are supported. + +The Keboola GitHub App has the client ID `Iv23liWeMeCpr1xBOVsj`. You can review the access you granted it, and +revoke it, at +[github.com/settings/connections/applications/Iv23liWeMeCpr1xBOVsj](https://github.com/settings/connections/applications/Iv23liWeMeCpr1xBOVsj). + +Setting this up takes two separate steps on GitHub, **in this order**: + +1. **Install** the app on your account or organisation + (`https://github.com/apps/keboola-custom-python/installations/new`) and choose which repositories it may read. + Repository selection happens here and nowhere else. +2. **Authorize** the component in the **Authorization** section of the configuration in Keboola. +3. Pick the repository from the **Repository** dropdown, which lists what the installation makes available. + +Installing and authorizing are independent. Authorizing does not install the app, and the authorization +screen offers no repository selection at all – so if you authorize without installing first, you receive a +valid token that can see no repositories and the **Repository** dropdown reports that none are available. To +change which repositories are available later, reconfigure the installation on GitHub; re-authorizing in +Keboola will not change it. + +Some organisations require an owner to approve the installation before it takes effect. + + +#### Choose "Only select repositories" + +The installation dialog offers **All repositories** or **Only select repositories**. Choose the second one and +list only the repositories this component needs. + +That dialog is the only place where the reach of the access token is decided, and narrowing it is the entire +reason to use OAuth rather than a personal access token. **All repositories** grants `Contents: Read-only` +across every repository in the account or organisation, including ones created later, and the token that +results does not expire – which is the same over-scoped, long-lived credential that a personal access token +was criticised for. Keboola cannot narrow this from its side; only the installation can. + +On a large organisation, **All repositories** also makes the **Repository** dropdown slow to load or unable to +load at all. A narrow selection avoids that. + +The token is always the intersection of what the app may read and what you can read yourself, so it never +reaches anything you could not already reach. The app requests `Contents: Read-only` and `Metadata: +Read-only`, and nothing else. + +Private git dependencies declared in `[tool.uv.sources]` authenticate with the same token and need no +credentials of their own. The repositories they live in have to be part of the installation's repository +selection as well, not just the repository holding the code. + +To change the selection later, reconfigure the installation on GitHub. Re-authorizing in Keboola does not +change it. + + ### Example: Running code saved in custom repository + template 🧩 As this might become a preferred way of running custom Python code in Keboola for many, we prepared a [simple example project](https://github.com/keboola/component-custom-python-example-repo-1), which help you with your first steps (and can also server you as a template for any of your future projects). diff --git a/component_config/configSchema.json b/component_config/configSchema.json index 75639a5..a4bf614 100644 --- a/component_config/configSchema.json +++ b/component_config/configSchema.json @@ -109,14 +109,22 @@ } }, "required": [ - "url", "auth" ], "properties": { "url": { "type": "string", "title": "Repository URL", - "propertyOrder": 70 + "propertyOrder": 70, + "options": { + "dependencies": { + "auth": [ + "none", + "pat", + "ssh" + ] + } + } }, "auth": { "type": "string", @@ -126,13 +134,16 @@ "enum": [ "none", "pat", - "ssh" + "ssh", + "oauth" ], "options": { + "tooltip": "The **GitHub (OAuth)** option is authorized in the **Authorization** section of this configuration. The repositories the component may read are selected on GitHub when the Keboola GitHub App is installed.", "enum_titles": [ "Public – None", "Private – Personal Access Token", - "Private – SSH Key" + "Private – SSH Key", + "Private – GitHub (OAuth)" ] }, "default": "none" @@ -158,6 +169,24 @@ } } }, + "repository": { + "type": "string", + "enum": [], + "format": "select", + "title": "Repository", + "propertyOrder": 105, + "options": { + "dependencies": { + "auth": "oauth" + }, + "tooltip": "Only repositories included in the Keboola GitHub App installation are listed. To make more of them available, change the repository selection of the installation on GitHub.", + "async": { + "label": "List Repositories", + "action": "listRepositories", + "cache": false + } + } + }, "branch": { "type": "string", "enum": [], @@ -169,7 +198,8 @@ "label": "List Branches", "action": "listBranches", "autoload": [ - "git.url" + "git.url", + "git.repository" ], "cache": false } diff --git a/src/component.py b/src/component.py index 9daff32..02dfa25 100644 --- a/src/component.py +++ b/src/component.py @@ -15,6 +15,7 @@ from keboola.component.exceptions import UserException from configuration import AuthEnum, Configuration, SourceEnum, VenvEnum, encrypted_keys +from github_api import GitHubApi from package_installer import PackageInstaller from source_file import FileHandler from source_git import GitHandler @@ -64,13 +65,29 @@ def __init__(self): f"in the configuration. Detail: {err}" ) from err + self.oauth_token = self._get_oauth_token() + + def _get_oauth_token(self) -> str | None: + """Access token issued by the OAuth broker, delivered outside "parameters" in the authorization + section. Returns None for configurations that do not use OAuth.""" + try: + credentials = self.configuration.oauth_credentials + except json.JSONDecodeError as err: + # unreadable broker credentials are a configuration problem, not an internal one + raise UserException( + "The stored GitHub authorization could not be read. Please authorize the component again " + "in the Authorization section of the configuration." + ) from err + + return credentials.data.get("access_token") if credentials else None + def run(self): if self.parameters.source == SourceEnum.CODE: base_path = Path(self.data_folder_path) script_filename = FileHandler.prepare_script_file(self.data_folder_path, self.parameters.code) else: base_path = Path(GitHandler.REPO_PATH).absolute() - git_handler = GitHandler(self.parameters.git) + git_handler = GitHandler(self.parameters.git, self.oauth_token) script_filename = git_handler.clone_repository() if self.parameters.venv == VenvEnum.BASE: @@ -87,7 +104,7 @@ def run(self): self.parameters.packages.insert(0, "keboola.component") PackageInstaller.install_packages(self.parameters.packages) else: - PackageInstaller.install_packages_for_repository(base_path) + PackageInstaller.install_packages_for_repository(base_path, git_handler.subprocess_env()) self._merge_user_parameters() @@ -147,18 +164,44 @@ def _merge_user_parameters(self): # remove code config_data = self.configuration.config_data.copy() + # the authorization section carries the decrypted OAuth access token and the shared application + # secret, neither of which may reach the executed user script + config_data.pop("authorization", None) + # build config data and overwrite for the user script config_data["parameters"] = self.parameters.user_properties with open(Path(self.data_folder_path) / "config.json", "w+") as inp: json.dump(config_data, inp) + @sync_action("listRepositories") + def get_oauth_repositories(self): + """ + Returns the repositories the Keboola GitHub App is allowed to read. + This method is used to populate the repository dropdown in the UI. + """ + if not self.oauth_token: + raise UserException( + "GitHub authorization is missing. Please authorize the component in the Authorization " + "section of the configuration." + ) + + repositories = GitHubApi(self.oauth_token).list_installation_repositories() + if not repositories: + # authorizing does not install the app, so this is the expected state after authorizing alone + raise UserException( + "No repositories are available to the Keboola GitHub App. Install the app on the account " + "owning the repository and include that repository in the app's repository selection." + ) + + return repositories + @sync_action("listBranches") def get_repository_branches(self): """ Returns a list of branches in the git repository. This method is used to populate the branches dropdown in the UI. """ - git_handler = GitHandler(self.parameters.git) + git_handler = GitHandler(self.parameters.git, self.oauth_token) return git_handler.get_repository_branches() @sync_action("listFiles") @@ -167,7 +210,7 @@ def get_repository_files(self): Returns a list of branches in the git repository. This method is used to populate the branches dropdown in the UI. """ - git_handler = GitHandler(self.parameters.git) + git_handler = GitHandler(self.parameters.git, self.oauth_token) return git_handler.get_repository_files() diff --git a/src/configuration.py b/src/configuration.py index b3f9e3c..8ec6831 100644 --- a/src/configuration.py +++ b/src/configuration.py @@ -25,6 +25,7 @@ class AuthEnum(Enum): NONE = "none" PAT = "pat" SSH = "ssh" + OAUTH = "oauth" # the ssh_keys.keys.[#private,public] structure is based on Keboola's standard SSH keys UI element output structure @@ -42,12 +43,18 @@ class SSHKeysConfiguration: @dataclass class GitConfiguration: url: str = "" + repository: str = "" branch: str = "main" filename: str = "main.py" auth: AuthEnum = AuthEnum.NONE encrypted_token: str | None = None ssh_keys: SSHKeysConfiguration = field(default_factory=SSHKeysConfiguration) + @property + def repository_url(self) -> str: + """Repository to clone. OAuth configurations select it from a list instead of typing in a URL.""" + return self.repository if self.auth == AuthEnum.OAUTH else self.url + @dataclass class Configuration: diff --git a/src/github_api.py b/src/github_api.py new file mode 100644 index 0000000..3d3fb34 --- /dev/null +++ b/src/github_api.py @@ -0,0 +1,78 @@ +import json +import urllib.error +import urllib.request + +from keboola.component.exceptions import UserException + +API_BASE_URL = "https://api.github.com" +API_VERSION = "2022-11-28" +USER_AGENT = "keboola-custom-python-component" +PAGE_SIZE = 100 +REQUEST_TIMEOUT = 30 +# a user access token never reaches anywhere near this many pages, it is a runaway guard only +MAX_PAGES = 50 + + +class GitHubApi: + """Read-only client for the endpoints needed to list the repositories the app may read.""" + + def __init__(self, token: str, base_url: str = API_BASE_URL): + self.token = token + self.base_url = base_url.rstrip("/") + + def list_installation_repositories(self) -> list[dict]: + """Repositories of every app installation visible to the authorizing user, as dropdown options. + + The value is the clone URL, so that selecting a repository fills in the same thing the other + authentication methods expect to be typed in by hand. + """ + options = [] + for installation in self._get_all("/user/installations", "installations"): + path = f"/user/installations/{installation['id']}/repositories" + for repository in self._get_all(path, "repositories"): + options.append({"value": repository["clone_url"], "label": repository["full_name"]}) + + return options + + def _get_all(self, path: str, items_key: str) -> list[dict]: + items: list[dict] = [] + for page in range(1, MAX_PAGES + 1): + payload = self._get(f"{path}?per_page={PAGE_SIZE}&page={page}") + page_items = payload.get(items_key, []) + items.extend(page_items) + if not page_items or len(items) >= payload.get("total_count", 0): + break + + return items + + def _get(self, path: str) -> dict: + request = urllib.request.Request( + f"{self.base_url}{path}", + headers={ + "Authorization": f"Bearer {self.token}", + "Accept": "application/vnd.github+json", + "X-GitHub-Api-Version": API_VERSION, + # GitHub rejects requests without a User-Agent + "User-Agent": USER_AGENT, + }, + ) + + try: + with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT) as response: + return json.load(response) + except urllib.error.HTTPError as err: + raise UserException(self._explain_http_error(err.code, err.reason)) from err + except urllib.error.URLError as err: + raise UserException(f"Could not reach the GitHub API: {err.reason}") from err + + @staticmethod + def _explain_http_error(code: int, reason: str) -> str: + if code == 401: + return ( + "The GitHub authorization is no longer valid. Please authorize the component again in the " + "Authorization section of the configuration." + ) + if code == 403: + return "The GitHub API refused the request. The authorization may not have the required permissions." + + return f"The GitHub API returned an unexpected error: HTTP {code} {reason}" diff --git a/src/package_installer.py b/src/package_installer.py index 9e2b116..933248b 100644 --- a/src/package_installer.py +++ b/src/package_installer.py @@ -17,7 +17,7 @@ def install_packages(packages: list[str]): SubprocessRunner.run(args, MSG_OK, MSG_ERR) @staticmethod - def install_packages_for_repository(repository_path: Path): + def install_packages_for_repository(repository_path: Path, env: dict[str, str] | None = None): """ Install packages based on the given repository path. - If there is a pyproject.toml and a uv.lock file, run uv sync. @@ -25,6 +25,8 @@ def install_packages_for_repository(repository_path: Path): Args: repository_path (str): Path to the repository containing requirements.txt. + env: Environment for the installation. Carries the git credentials, without which + private git dependencies cannot be fetched. """ pyproject_file = repository_path / "pyproject.toml" uv_lock_file = repository_path / "uv.lock" @@ -46,4 +48,4 @@ def install_packages_for_repository(repository_path: Path): logging.info("No dependencies file found") return - SubprocessRunner.run(args, MSG_OK, MSG_ERR) + SubprocessRunner.run(args, MSG_OK, MSG_ERR, env) diff --git a/src/source_git.py b/src/source_git.py index 5fd5fba..ee5faac 100644 --- a/src/source_git.py +++ b/src/source_git.py @@ -9,42 +9,53 @@ from configuration import AuthEnum, GitConfiguration +GITHUB_HOSTS = ("github.com", "www.github.com") +OAUTH_GIT_USERNAME = "x-access-token" +OAUTH_TOKEN_ENV = "GIT_OAUTH_TOKEN" +# git runs this helper whenever it needs a password. Reading the token from the environment keeps it out +# of the command line, out of the cloned repository's .git/config and out of the user script's environment. +ASKPASS_SCRIPT = f'#!/bin/sh\nprintf "%s" "${OAUTH_TOKEN_ENV}"\n' + class GitHandler: REPO_PATH = "repo_clone" - def __init__(self, git_cfg: GitConfiguration): + def __init__(self, git_cfg: GitConfiguration, oauth_token: str | None = None): # add path for absolute imports to start at the cloned repository root level sys.path.append(str(Path(__file__).parent.parent / GitHandler.REPO_PATH)) - self.env = os.environ.copy() + # only the git-specific overrides; the full environment is resolved at call time so that + # changes made after the clone (the virtual environment selection) are not lost + self.git_env: dict[str, str] = {} self.git_cfg = git_cfg + self.repo_url = git_cfg.repository_url self.repo_auth_url = None # ‼️ NEVER EVER INCLUDE THIS VARIABLE IN LOGGING OUTPUT ‼️ - if not self.git_cfg.url: - raise UserException("Git repository URL is required") + if not self.repo_url: + raise UserException( + "Please select a repository" if git_cfg.auth == AuthEnum.OAUTH else "Git repository URL is required" + ) if self.git_cfg.auth == AuthEnum.PAT: self._set_up_token_auth() + elif self.git_cfg.auth == AuthEnum.OAUTH: + self._set_up_oauth_auth(oauth_token) - repo_url = self.git_cfg.url - if repo_url.startswith("git@") or repo_url.startswith("ssh://"): + if self.repo_url.startswith("git@") or self.repo_url.startswith("ssh://"): self._set_up_ssh_command() # do not ask for credentials when git authentication fails - self.env["GIT_TERMINAL_PROMPT"] = "0" + self.git_env["GIT_TERMINAL_PROMPT"] = "0" def _set_up_token_auth(self) -> None: if not self.git_cfg.encrypted_token: raise UserException("No personal access token provided") - if not self.git_cfg.url.startswith("https://"): + if not self.repo_url.startswith("https://"): raise UserException("PAT authentication is only supported for HTTPS URLs") - self.repo_auth_url = self.git_cfg.url.replace( - "https://", f"https://x-token-auth:{self.git_cfg.encrypted_token}@" - ) - self._set_up_netrc(self.git_cfg.url, self.git_cfg.encrypted_token) + self.repo_auth_url = self.repo_url.replace("https://", f"https://x-token-auth:{self.git_cfg.encrypted_token}@") + self._set_up_netrc(self.repo_url, self.git_cfg.encrypted_token) logging.info("Git token authentication set up for HTTPS URL.") @staticmethod @@ -58,6 +69,30 @@ def _set_up_netrc(repo_url: str, token: str) -> None: f.write(entry) os.chmod(netrc_path, 0o600) + def _set_up_oauth_auth(self, oauth_token: str | None) -> None: + if not oauth_token: + raise UserException( + "GitHub authorization is missing. Please authorize the component in the Authorization " + "section of the configuration." + ) + + parsed = urlparse(self.repo_url) + if parsed.scheme != "https" or parsed.hostname not in GITHUB_HOSTS: + raise UserException("GitHub authorization is only supported for https://github.com repository URLs") + + # only the username goes into the URL, the token itself is supplied by the askpass helper + self.repo_auth_url = self.repo_url.replace("https://", f"https://{OAUTH_GIT_USERNAME}@") + self.git_env[OAUTH_TOKEN_ENV] = oauth_token + self.git_env["GIT_ASKPASS"] = str(self._write_askpass_helper()) + logging.info("Git OAuth authentication set up for GitHub URL.") + + @staticmethod + def _write_askpass_helper() -> Path: + askpass_path = Path("~/.git_askpass.sh").expanduser() + askpass_path.write_text(ASKPASS_SCRIPT) + os.chmod(askpass_path, 0o700) + return askpass_path + def _set_up_ssh_command(self) -> None: if not self.git_cfg.ssh_keys.keys.encrypted_private: if self.git_cfg.auth == AuthEnum.SSH: @@ -88,7 +123,31 @@ def _set_up_ssh_command(self) -> None: os.chmod(ssh_key_path, 0o600) ssh_command.extend(["-i", str(ssh_key_path)]) - self.env["GIT_SSH_COMMAND"] = " ".join(ssh_command) + self.git_env["GIT_SSH_COMMAND"] = " ".join(ssh_command) + + def _explain_error(self, error_msg: str) -> str: + """Append an actionable hint to git errors whose raw wording does not point at the actual cause.""" + if "Permission denied" in error_msg or "publickey" in error_msg: + return f"{error_msg}. Please check SSH key configuration or use HTTPS URL." + + # GitHub answers with "not found" for repositories the app cannot see, so that it does not + # disclose their existence. The usual cause is a missing or incomplete app installation. + if self.git_cfg.auth == AuthEnum.OAUTH and "not found" in error_msg.lower(): + return ( + f"{error_msg}. The repository is not available to the Keboola GitHub App. Make sure the app " + "is installed on the account owning the repository and that this repository is included in " + "the app's repository selection." + ) + + return error_msg + + def subprocess_env(self) -> dict[str, str]: + """Environment for a subprocess that needs to reach the repository, credentials included. + + Also used for the dependency installation, so that private git dependencies declared in the + repository authenticate with the same credentials as the clone itself. + """ + return {**os.environ, **self.git_env} def clone_repository(self, sync_action=False) -> Path: """ @@ -99,7 +158,7 @@ def clone_repository(self, sync_action=False) -> Path: """ branch = self.git_cfg.branch or "main" - logging.info("Cloning git repository: %s", self.git_cfg.url) + logging.info("Cloning git repository: %s", self.repo_url) try: clone_args = ["git", "clone"] @@ -107,21 +166,19 @@ def clone_repository(self, sync_action=False) -> Path: if branch: clone_args.extend(["--branch", branch]) - clone_args.extend([self.repo_auth_url or self.git_cfg.url, GitHandler.REPO_PATH]) + clone_args.extend([self.repo_auth_url or self.repo_url, GitHandler.REPO_PATH]) process = subprocess.Popen( clone_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - env=self.env, + env=self.subprocess_env(), ) _, stderr = process.communicate() if process.returncode != 0: error_msg = stderr.decode() if stderr else "Unknown git clone error" - if "Permission denied" in error_msg or "publickey" in error_msg: - error_msg += ". Please check SSH key configuration or use HTTPS URL." - raise UserException(f"Failed to clone git repository: {error_msg}") + raise UserException(f"Failed to clone git repository: {self._explain_error(error_msg)}") logging.info("Successfully cloned repository") @@ -150,18 +207,18 @@ def get_repository_branches(self): try: branches_args = ["git", "ls-remote", "--heads"] - branches_args.append(self.repo_auth_url or self.git_cfg.url) + branches_args.append(self.repo_auth_url or self.repo_url) process = subprocess.Popen( branches_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, - env=self.env, + env=self.subprocess_env(), ) stdout, stderr = process.communicate() if process.returncode != 0: - raise UserException(f"Failed to get branches: {stderr.decode()}") + raise UserException(f"Failed to get branches: {self._explain_error(stderr.decode())}") branches = [line.strip().split("refs/heads/")[-1] for line in stdout.decode().splitlines() if line.strip()] return [{"value": b, "label": b} for b in branches] diff --git a/src/subprocess_runner.py b/src/subprocess_runner.py index f6e68f2..322231d 100644 --- a/src/subprocess_runner.py +++ b/src/subprocess_runner.py @@ -63,6 +63,7 @@ def run( args: list[str], ok_message: str = "Command finished successfully.", err_message: str = "Command failed.", + env: dict[str, str] | None = None, ): logging.debug("Running command: %s", " ".join(args)) process = subprocess.Popen( @@ -70,6 +71,7 @@ def run( stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + env=env, ) stderr_output: deque[str] = deque(maxlen=MAX_STDERR_LINES) diff --git a/tests/test_component.py b/tests/test_component.py index cbe119e..ca19cbb 100644 --- a/tests/test_component.py +++ b/tests/test_component.py @@ -1,14 +1,27 @@ +import io import json import os import tempfile import unittest +import urllib.error +from pathlib import Path import mock from freezegun import freeze_time from keboola.component.exceptions import UserException from component import Component -from configuration import Configuration, SourceEnum, VenvEnum +from configuration import AuthEnum, Configuration, GitConfiguration, SourceEnum, VenvEnum +from github_api import GitHubApi +from package_installer import PackageInstaller +from source_git import GitHandler + + +def api_response(payload: dict): + """A urlopen context manager yielding the given payload as a JSON body.""" + response = mock.MagicMock() + response.__enter__.return_value = io.BytesIO(json.dumps(payload).encode()) + return response class TestComponent(unittest.TestCase): @@ -128,6 +141,306 @@ def test_valid_configuration_is_parsed_unchanged(self): self.assertEqual(component.parameters.code, "print('hello')") +class TestOAuthAuthentication(unittest.TestCase): + """The OAuth access token must reach git without ever appearing in the command line. + + The component executes arbitrary user code, so the token is passed through an askpass helper that + reads it from the environment of the git subprocess only. + """ + + def setUp(self): + home = tempfile.TemporaryDirectory() + self.addCleanup(home.cleanup) + home_patch = mock.patch.dict(os.environ, {"HOME": home.name}) + home_patch.start() + self.addCleanup(home_patch.stop) + + @staticmethod + def _git_cfg(url: str = "https://github.com/keboola/example.git") -> GitConfiguration: + return GitConfiguration(repository=url, auth=AuthEnum.OAUTH) + + def test_missing_repository_asks_for_a_selection(self): + """The repository is picked from a dropdown here, so asking for a URL would be confusing.""" + with self.assertRaises(UserException) as context: + GitHandler(GitConfiguration(auth=AuthEnum.OAUTH), "secret-token") + self.assertIn("select a repository", str(context.exception)) + + def test_missing_url_still_asks_for_a_url(self): + """The wording for the other authentication methods is unchanged.""" + with self.assertRaises(UserException) as context: + GitHandler(GitConfiguration(auth=AuthEnum.PAT)) + self.assertIn("URL is required", str(context.exception)) + + def test_missing_token_raises_user_exception(self): + """An unauthorized configuration must fail with an actionable message, not with a git error.""" + with self.assertRaises(UserException) as context: + GitHandler(self._git_cfg(), None) + self.assertIn("GitHub authorization is missing", str(context.exception)) + + def test_non_github_url_raises_user_exception(self): + """The token is only valid for github.com, other hosts must be rejected up front.""" + with self.assertRaises(UserException) as context: + GitHandler(self._git_cfg("https://gitlab.com/keboola/example.git"), "secret-token") + self.assertIn("github.com", str(context.exception)) + + def test_token_is_not_part_of_the_clone_url(self): + """The clone URL carries the username only, so the token cannot leak via argv or .git/config.""" + handler = GitHandler(self._git_cfg(), "secret-token") + self.assertEqual(handler.repo_auth_url, "https://x-access-token@github.com/keboola/example.git") + + def test_token_is_passed_through_the_askpass_helper(self): + """The helper is executable and reads the token from the environment instead of embedding it.""" + handler = GitHandler(self._git_cfg(), "secret-token") + self.assertEqual(handler.git_env["GIT_OAUTH_TOKEN"], "secret-token") + + askpass_path = Path(handler.git_env["GIT_ASKPASS"]) + self.assertTrue(os.access(askpass_path, os.X_OK)) + self.assertNotIn("secret-token", askpass_path.read_text()) + + def test_missing_installation_is_explained(self): + """GitHub reports an unreachable repository as "not found", which hides the real cause.""" + handler = GitHandler(self._git_cfg(), "secret-token") + explained = handler._explain_error("remote: Repository not found.") + self.assertIn("Keboola GitHub App", explained) + self.assertIn("repository selection", explained) + + def test_unrelated_errors_are_not_annotated(self): + handler = GitHandler(self._git_cfg(), "secret-token") + self.assertEqual(handler._explain_error("fatal: could not read from remote"), "fatal: could not read from remote") + + def test_token_stays_out_of_the_process_environment(self): + """The executed user script inherits os.environ, so the token must never be put there.""" + handler = GitHandler(self._git_cfg(), "secret-token") + self.assertNotIn("GIT_OAUTH_TOKEN", os.environ) + self.assertEqual(handler.subprocess_env()["GIT_OAUTH_TOKEN"], "secret-token") + + def test_subprocess_env_reflects_later_environment_changes(self): + """The virtual environment is chosen after the clone, so the env cannot be a stale snapshot.""" + handler = GitHandler(self._git_cfg(), "secret-token") + with mock.patch.dict(os.environ, {"UV_PROJECT_ENVIRONMENT": "/code/repo_clone/.venv"}): + env = handler.subprocess_env() + self.assertEqual(env["UV_PROJECT_ENVIRONMENT"], "/code/repo_clone/.venv") + self.assertEqual(env["GIT_OAUTH_TOKEN"], "secret-token") + + def test_other_auth_methods_are_untouched(self): + """A configuration that does not use OAuth must not gain any OAuth environment.""" + handler = GitHandler(GitConfiguration(url="https://github.com/keboola/example.git")) + self.assertIsNone(handler.repo_auth_url) + self.assertNotIn("GIT_ASKPASS", handler.git_env) + self.assertNotIn("GIT_OAUTH_TOKEN", handler.git_env) + + def test_ssh_hint_is_preserved(self): + """The pre-existing hint for SSH failures must keep working for non-OAuth configurations.""" + handler = GitHandler(GitConfiguration(url="git@github.com:keboola/example.git", auth=AuthEnum.NONE)) + self.assertIn("SSH key configuration", handler._explain_error("Permission denied (publickey).")) + + +class TestAuthorizationSectionIsNotExposed(unittest.TestCase): + """The config.json handed to the user script must not contain the decrypted OAuth credentials. + + Besides the user's own access token, the authorization section also carries the shared application + secret of the Keboola GitHub App, which must never be readable by the executed script. + """ + + CONFIG_DATA = { + "parameters": {"source": "code", "venv": "base", "user_properties": {"debug": True}}, + "authorization": { + "oauth_api": { + "credentials": { + "id": "main", + "#data": '{"access_token": "secret-token"}', + "appKey": "client-id", + "#appSecret": "app-secret", + } + } + }, + } + + def setUp(self): + datadir = tempfile.TemporaryDirectory() + self.addCleanup(datadir.cleanup) + self.config_path = Path(datadir.name) / "config.json" + self.config_path.write_text(json.dumps(self.CONFIG_DATA)) + with mock.patch.dict(os.environ, {"KBC_DATADIR": datadir.name}): + self.component = Component() + + def test_access_token_is_read_from_the_authorization_section(self): + self.assertEqual(self.component.oauth_token, "secret-token") + + def test_unreadable_credentials_raise_user_exception(self): + """Broker credentials that are not valid JSON must not surface as an internal error.""" + datadir = tempfile.TemporaryDirectory() + self.addCleanup(datadir.cleanup) + config_data = dict(self.CONFIG_DATA) + config_data["authorization"] = {"oauth_api": {"credentials": {"id": "main", "#data": "access_token=abc"}}} + (Path(datadir.name) / "config.json").write_text(json.dumps(config_data)) + + with mock.patch.dict(os.environ, {"KBC_DATADIR": datadir.name}): + with self.assertRaises(UserException) as context: + Component() + self.assertIn("could not be read", str(context.exception)) + + def test_authorization_is_stripped_from_the_script_config(self): + self.component._merge_user_parameters() + + written = self.config_path.read_text() + self.assertNotIn("authorization", json.loads(written)) + self.assertNotIn("secret-token", written) + self.assertNotIn("app-secret", written) + + def test_user_properties_are_still_written(self): + """Stripping the credentials must not disturb what the script actually needs.""" + self.component._merge_user_parameters() + + self.assertEqual(json.loads(self.config_path.read_text())["parameters"], {"debug": True}) + + +class TestGitHubApi(unittest.TestCase): + """Listing repositories has to work across several installations and several pages.""" + + def test_repositories_from_all_installations_are_listed(self): + responses = [ + api_response({"total_count": 2, "installations": [{"id": 1}, {"id": 2}]}), + api_response( + {"total_count": 1, "repositories": [{"full_name": "acme/first", "clone_url": "https://gh/first.git"}]} + ), + api_response( + {"total_count": 1, "repositories": [{"full_name": "acme/second", "clone_url": "https://gh/second.git"}]} + ), + ] + with mock.patch("github_api.urllib.request.urlopen", side_effect=responses): + options = GitHubApi("secret-token").list_installation_repositories() + + self.assertEqual( + options, + [ + {"value": "https://gh/first.git", "label": "acme/first"}, + {"value": "https://gh/second.git", "label": "acme/second"}, + ], + ) + + def test_paginated_results_are_collected(self): + first_page = [{"full_name": f"acme/repo-{i}", "clone_url": f"https://gh/repo-{i}.git"} for i in range(100)] + responses = [ + api_response({"total_count": 1, "installations": [{"id": 1}]}), + api_response({"total_count": 101, "repositories": first_page}), + api_response({"total_count": 101, "repositories": [{"full_name": "acme/last", "clone_url": "https://gh/l"}]}), + ] + with mock.patch("github_api.urllib.request.urlopen", side_effect=responses): + options = GitHubApi("secret-token").list_installation_repositories() + + self.assertEqual(len(options), 101) + self.assertEqual(options[-1]["label"], "acme/last") + + def test_request_is_authenticated(self): + response = api_response({"total_count": 0, "installations": []}) + with mock.patch("github_api.urllib.request.urlopen", side_effect=[response]) as urlopen: + GitHubApi("secret-token").list_installation_repositories() + + headers = {key.lower(): value for key, value in urlopen.call_args.args[0].header_items()} + self.assertEqual(headers["authorization"], "Bearer secret-token") + self.assertIn("user-agent", headers) + + def test_revoked_authorization_is_explained(self): + """A revoked authorization must tell the user to re-authorize, not show a bare HTTP 401.""" + error = urllib.error.HTTPError("https://api.github.com/user/installations", 401, "Unauthorized", {}, None) + with mock.patch("github_api.urllib.request.urlopen", side_effect=error): + with self.assertRaises(UserException) as context: + GitHubApi("secret-token").list_installation_repositories() + + self.assertIn("authorize the component again", str(context.exception)) + + +class TestListRepositoriesAction(unittest.TestCase): + """The dropdown is where a missing app installation shows up before a job is ever run.""" + + def _component(self, authorized: bool) -> Component: + datadir = tempfile.TemporaryDirectory() + self.addCleanup(datadir.cleanup) + # "run" keeps the sync_action decorator from swallowing exceptions into exit(1) + config_data = {"action": "run", "parameters": {"source": "git", "venv": "base", "user_properties": {}}} + if authorized: + credentials = {"id": "main", "#data": '{"access_token": "secret-token"}'} + config_data["authorization"] = {"oauth_api": {"credentials": credentials}} + (Path(datadir.name) / "config.json").write_text(json.dumps(config_data)) + + with mock.patch.dict(os.environ, {"KBC_DATADIR": datadir.name}): + return Component() + + def test_unauthorized_configuration_is_reported(self): + with self.assertRaises(UserException) as context: + self._component(authorized=False).get_oauth_repositories() + self.assertIn("GitHub authorization is missing", str(context.exception)) + + def test_missing_installation_is_reported(self): + component = self._component(authorized=True) + response = api_response({"total_count": 0, "installations": []}) + with mock.patch("github_api.urllib.request.urlopen", side_effect=[response]): + with self.assertRaises(UserException) as context: + component.get_oauth_repositories() + + self.assertIn("No repositories are available", str(context.exception)) + + +class TestRepositoryUrlResolution(unittest.TestCase): + """OAuth configurations carry the repository in "repository", the other methods in "url".""" + + def test_oauth_uses_the_selected_repository(self): + cfg = GitConfiguration(url="https://github.com/acme/typed.git", auth=AuthEnum.OAUTH, + repository="https://github.com/acme/picked.git") + self.assertEqual(cfg.repository_url, "https://github.com/acme/picked.git") + + def test_other_methods_use_the_typed_url(self): + cfg = GitConfiguration(url="https://github.com/acme/typed.git", auth=AuthEnum.PAT, + repository="https://github.com/acme/picked.git") + self.assertEqual(cfg.repository_url, "https://github.com/acme/typed.git") + + +class TestDependencyInstallationCredentials(unittest.TestCase): + """A repository's private git dependencies must authenticate with the credentials of the clone. + + `uv sync` shells out to git, which offers no credentials of its own, so without an explicit + environment the fetch fails with "could not read Username for https://github.com". + """ + + def _repository(self, *files: str) -> Path: + repo = tempfile.TemporaryDirectory() + self.addCleanup(repo.cleanup) + self.addCleanup(os.chdir, os.getcwd()) + repo_path = Path(repo.name) + for name in files: + (repo_path / name).write_text("") + return repo_path + + def test_environment_is_forwarded_to_uv_sync(self): + repo_path = self._repository("pyproject.toml", "uv.lock") + + with mock.patch("package_installer.SubprocessRunner.run") as run: + PackageInstaller.install_packages_for_repository(repo_path, {"GIT_OAUTH_TOKEN": "secret-token"}) + + args = run.call_args.args + self.assertEqual(args[0], ["uv", "sync", "--inexact"]) + self.assertEqual(args[3], {"GIT_OAUTH_TOKEN": "secret-token"}) + + def test_environment_is_forwarded_to_requirements_install(self): + """requirements.txt can reference private git URLs just as pyproject.toml can.""" + repo_path = self._repository("requirements.txt") + + with mock.patch("package_installer.SubprocessRunner.run") as run: + PackageInstaller.install_packages_for_repository(repo_path, {"GIT_OAUTH_TOKEN": "secret-token"}) + + self.assertEqual(run.call_args.args[3], {"GIT_OAUTH_TOKEN": "secret-token"}) + + def test_installation_without_credentials_still_works(self): + """Configurations that need no credentials must keep inheriting the process environment.""" + repo_path = self._repository("pyproject.toml", "uv.lock") + + with mock.patch("package_installer.SubprocessRunner.run") as run: + PackageInstaller.install_packages_for_repository(repo_path) + + self.assertIsNone(run.call_args.args[3]) + + if __name__ == "__main__": # import sys;sys.argv = ['', 'Test.testName'] unittest.main()