Locally running Prowler app on docker compose unable to locate credentials

[derkmingfong]

Issue with using both IAM Role and using credentials

I tried running the following commands

curl -LO https://raw.githubusercontent.com/prowler-cloud/prowler/refs/heads/master/docker-compose.yml
curl -LO https://raw.githubusercontent.com/prowler-cloud/prowler/refs/heads/master/.env
docker compose up -d

Signed up at http://localhost:3000/

I have set up my CloudFormation stack based on the instructions from https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/prowler-app/

But still the error persists.

Any advice is appreciated

[jfagoagas]

Hi @derkmingfong, thanks for reaching out!

To help you effectively, I first need to understand exactly what you're trying to achieve.

Are you attempting to add a new AWS provider using an IAM Role? If so, are you explicitly setting the AWS access and secret keys, or relying on the SDK to use default credentials?

If you're relying on the SDK defaults, you'll need to ensure that the .aws folder is properly mapped in your docker-compose file, adding the following to the worker service:

worker:
  volumes:
    - "${HOME}/.aws:/home/prowler/.aws:ro"

This allows the worker containers to access the AWS credentials from your local environment.

Let me know a bit more about your setup so I can guide you further!

Thanks for using Prowler 🚀

[derkmingfong]

Noted I will test this out later.

Meanwhile I have been trying to use a different approach to run Prowler from github instead of using docker compose method

From GitHub
Requirements

git installed.
poetry v2 installed: poetry installation.
npm installed: npm installation.
Docker Compose installed: https://docs.docker.com/compose/install/.
Commands to run the API

git clone https://github.com/prowler-cloud/prowler
cd prowler/api
poetry install
eval $(poetry env activate)
set -a
source .env
docker compose up postgres valkey -d
cd src/backend

Starting from this step "python manage.py migrate --database admin", I kept getting errors with DJANGO

I have attached the full logs of the steps used

$ set -a
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api (master)
$ source .env
bash: devel]: command not found
bash: human_readable]: command not found
bash: False]: command not found
bash: postgres-db]: command not found
bash: valkey]: command not found
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api (master)
$ cd src/backend
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api/src/backend (master)
$ python manage.py migrate --database admin
Traceback (most recent call last):
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\manage.py", line 30, in
main()
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\manage.py", line 26, in main
execute_from_command_line(sys.argv)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management_init_.py", line 442, in execute_from_command_line
utility.execute()
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management_init_.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 459, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 107, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\commands\migrate.py", line 101, in handle
self.check(databases=[database])
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 486, in check
all_issues = checks.run_checks(
^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\registry.py", line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\urls.py", line 16, in check_url_config
return check_resolver(resolver)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\urls.py", line 26, in check_resolver
return check_method()
^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 531, in check
for pattern in self.url_patterns:
^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\utils\functional.py", line 47, in get
res = instance.dict[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 718, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\utils\functional.py", line 47, in get
res = instance.dict[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 711, in urlconf_module
return import_module(self.urlconf_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\AppData\Local\Programs\Python\Python312\Lib\importlib_init_.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 999, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\config\urls.py", line 6, in
path("api/v1/", include("api.v1.urls")),
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\conf.py", line 39, in include
urlconf_module = import_module(urlconf_module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\AppData\Local\Programs\Python\Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in find_and_load_unlocked
File "", line 935, in load_unlocked
File "", line 999, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\api\v1\urls.py", line 5, in
from api.v1.views import (
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\api\v1\views.py", line 53, in
from tasks.beat import schedule_provider_scan
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\beat.py", line 6, in
from tasks.tasks import perform_scheduled_scan_task
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\tasks.py", line 13, in
from tasks.jobs.export import (
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\jobs\export.py", line 49, in
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365 import (
ModuleNotFoundError: No module named 'prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365'
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api/src/backend (master)
$ python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
ttJPymlC6k4yfnebBS_0yogwoP2iC9GKYgE8y1gS7X4=
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api/src/backend (master)
$ export SECRETS_ENCRYPTION_KEY=ttJPymlC6k4yfnebBS_0yogwoP2iC9GKYgE8y1gS7X4=
(python_prowler_3.12)
derkming.fong@RWMYOA-LTP-020 MINGW64 ~/prowler_app/prowler/api/src/backend (master)
$ python manage.py migrate --database admin
Traceback (most recent call last):
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\manage.py", line 30, in
main()
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\manage.py", line 26, in main
execute_from_command_line(sys.argv)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management_init.py", line 442, in execute_from_command_line
utility.execute()
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management_init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 459, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 107, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\commands\migrate.py", line 101, in handle
self.check(databases=[database])
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\management\base.py", line 486, in check
all_issues = checks.run_checks(
^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\registry.py", line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\urls.py", line 16, in check_url_config
return check_resolver(resolver)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\core\checks\urls.py", line 26, in check_resolver
return check_method()
^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 531, in check
for pattern in self.url_patterns:
^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\utils\functional.py", line 47, in get
res = instance.dict[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 718, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\utils\functional.py", line 47, in get
res = instance.dict[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\resolvers.py", line 711, in urlconf_module
return import_module(self.urlconf_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\AppData\Local\Programs\Python\Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 999, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\config\urls.py", line 6, in
path("api/v1/", include("api.v1.urls")),
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\prowler_app\prowler\python_prowler_3.12\Lib\site-packages\django\urls\conf.py", line 39, in include
urlconf_module = import_module(urlconf_module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\derkming.fong\AppData\Local\Programs\Python\Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\api\v1\urls.py", line 5, in
from api.v1.views import (
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\api\v1\views.py", line 53, in
from tasks.beat import schedule_provider_scan
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\beat.py", line 6, in
from tasks.tasks import perform_scheduled_scan_task
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\tasks.py", line 13, in
from tasks.jobs.export import (
File "C:\Users\derkming.fong\prowler_app\prowler\api\src\backend\tasks\jobs\export.py", line 49, in
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365 import (
ModuleNotFoundError: No module named 'prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365'

[jfagoagas]

We are going to review it because I think some step is not correct or up to date. In the meantime I can say that you must use the .env file from the root folder instead of the one in the prowler/api because that is not present anymore in the master branch.

[jfagoagas]

Also, please pull latest changes from prowler:master because there was an issue with some dependencies in the .lock file.

[derkmingfong]

Hi @jfagoagas Thanks for the update.

Just to clarify, you are referring to the poetry.lock file under the prowler/api? I need to pull the latest changes from branch prowler:master.

Also I have tried the .env file when using the docker compose method in your READ.ME file and it still does not allow connection to my AWS account.

Perhaps my understanding of the steps are not clear. looking forward to the review of the steps needed to run the prowler app locally.

[jfagoagas]

Just to clarify, you are referring to the poetry.lock file under the prowler/api? I need to pull the latest changes from branch prowler:master.

Correct!

Also I have tried the .env file when using the docker compose method in your READ.ME file and it still does not allow connection to my AWS account.
Perhaps my understanding of the steps are not clear. looking forward to the review of the steps needed to run the prowler app locally.

I think at this point it'd be better if you can talk directly with the team via our Slack Community at goto.prowler.com/slack. Please reach any of us there anytime to have a quick call. Thanks!

[derkmingfong]

@DMF194 tagging to self for reference