Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from http://github.com/IBM/detect-secrets-stream below:

IBM/detect-secrets-stream: Detect Secrets Stream focuses on the detection of leaks by scanning every push to GitHub Enterprise in a transparent, non-blocking manner, and reporting its findings to a downstream remediation team for follow-up.

Detect Secrets Stream is a server tool which ingests metadata of all (public repositories by default, private repositories are opt-in only) git pushes on your company's GitHub Enterprise server. For each push, it scans the push contents for secrets. Once found and verified, secrets metadata will be stored in a database, and the raw secret will be stored in Vault.

There is a companion Admin tool which enables org admins to:

• Opt-in their private repository for scanning
• Add security folks to be notified of leaked tokens

Under the hood, the server tool uses the developer tool to scan for secrets. Read IBM/detect-secrets for more info about developer tool.

• python3
  • If you are using macOs, you can use pyenv to install Python 3: brew install pyenv; pyenv install 3.8.5;
  • Optionally, you can set your system's default Python to Python 3 pyenv global 3.8.5. Then restart your shell. Run python --version to validate the default Python version is 3.
• docker https://docs.docker.com/get-docker/
• skaffold, v1.12.1 and above
• kustomize, v3.8.1 and above. Do NOT use the version bundled with kubectl, as it does not support some options use by this project (e.g. replicas).
• container-structure-test, used for docker image validation - installation.
• pipenv, used to manage all Python dependencies.

You can install all the tools except docker and python 3 with one liner below

brew install kustomize skaffold container-structure-test pipenv

• Navigate into the cloned repo
• Start the pipenv shell with pipenv shell
• Install Python dependencies with pipenv install --dev
• Initialize the pre-commit tool with pre-commit install

Example secrets can be found in secrets.template (more docs coming). For your local dev environment, some secrets are auto-generated. For prod environments, you will need to supply the real secrets.

To set up your local dev secrets, first run ./kustomize_envs/dev/gen-secret.sh. It will create two hidden folders under /kustomize_envs/dev/:

• secret_generated/ containing automatically-generated secrets. Secrets get regenerated when re-running gen-secret.sh.
• secret_manual/ containing manually-entered secrets. Running gen-secret.sh will generate a template for secrets under this folder. You do need to manually update each secret to match your environment. Re-running gen-secret.sh will not overwrite the files if they are non-empty.

This table contains information on what the values of the manually-entered secrets should be set to:

Besides the secrets mentioned from Local dev secrets, for production environments, you also need to prepare secrets which are auto-generated in the dev environment.

This table contains information on what the values of the dev secrets should be set to:

You don't need to unlock secrets when running unit tests.

This provides faster feedback if you are just writing code for a module.

# The part after last dash (-) corresponding to folder name under detect_secrets_stream
# For example, run unit test for just files under bp_lookup
make test-unit-bp_lookup
# Run unit test for just files under pi_cleaner
make test-unit-pi_cleaner

This requires a personal or staging environment. See kustomize_envs/dev/README.md for more details.

This repo has provided a utility module which enables an admin to do many routine tasks. The utility is invoked though python -m detect_secrets_stream.util.secret_util

Running the utility requires several environment variables. An example environment variable file (.env.example) has been provided.

From a fish shell, you can do something like below

cp .env.example .env.prod
env (grep -v '^#' .env.prod | xargs -n1) python -m detect_secrets_stream.util.secret_util --help

Based on id

python -m detect_secrets_stream.util.secret_util decrypt-token-by-id [token_id]

Based on UUID

python -m detect_secrets_stream.util.secret_util decrypt-token-by-uuid [uuid]

python -m detect_secrets_stream.util.secret_util get-org-admins [ORG_NAME]

# export require env vars

# macos
python -m detect_secrets_stream.util.secret_util backfill --size=10000 --from $(date -j -f "%a %b %d %T %Z %Y" "Wed Sep 11 00:00:00 EDT 2019" +"%s") --to $(date -j -
f "%a %b %d %T %Z %Y" "Wed Sep 12 00:00:00 EDT 2019" +"%s")

# linux
python -m detect_secrets_stream.util.secret_util backfill --size=10000 --from $(date -d '06/12/2012 07:21:22' +"%s") --to $(date -d '06/12/2012 08:21:22' +"%s")

Manually add a commit to the diff-scan queue.

Note: must set KAFKA_CLIENT_ID , GD_KAFKA_CONF environment variables.

KAFKA_CLIENT_ID is the name of the Kafka client used for manual ingestion. It can be anything, such as manual-ingest GD_KAFKA_CONF points to the Kafka configuration file. The production config is stored under kustomize_envs/prod-secrets/secret/kafka.conf. The one below is an example of what should be contained in the config file.

[kafka]
brokers_sasl = my_sasl1.us-east.containers.appdomain.cloud:9000,my_sasl2.us-east.containers.appdomain.cloud:9000,my_sasl3.us-east.containers.appdomain.cloud:9000
api_key = my_api_key

Sample usage:

# export require env vars
python -m detect_secrets_stream.util.secret_util ingest-commit -r <repo> -c <commit>

More options: running the command above with --help will reveal help info on more options. For example, if you know the branch and repository's visibility, you can also supply these. By default, it assumes the commit is from the master branch and the repository's visibility is public.

python -m detect_secrets_stream.util.secret_util --help

The mechanism for removing old secrets is a cronjob which cleans PI on a daily basis.

It removes the following PI for all tokens that have been remediated for over seven days:

• Owner email
• Secret
• Encrypted secret
• Other factors
• Author name
• Author email
• Pusher username
• Pusher email
• Committer name
• Committer email
• Repo slug
• Location url

Every four hours, a cronjob checks if all live tokens in the database are still live and updates their statuses accordingly.

CREATE ROLE scan_worker_role;
GRANT SELECT, INSERT ON ALL TABLES IN SCHEMA public TO scan_worker_role;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO scan_worker_role;
GRANT TRUNCATE ON TABLE public.vmt_report TO scan_worker_role;

CREATE ROLE vmt_role WITH LOGIN;
GRANT CONNECT ON DATABASE <db_name> TO vmt_role;
GRANT USAGE ON SCHEMA public TO vmt_role;
GRANT SELECT ON public.vmt_report TO vmt_role;

CREATE USER vmt_user WITH IN GROUP vmt_role PASSWORD [redacted]

See the additional docs for more information:

• Detect Secrets Stream FAQ
• Admin Tool FAQ
• Adding a New Detector
• Running a local end 2 end test
• Operation Guide

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4