AWS Lambda is a powerful tool available from Amazon Web Services. It offers Serverless functions that cost nothing when not used, and then fractions of a penny when invoked.
A common requirement for a Lambda function is to retrieve information from a secure location, and use it to access a database, file store or other secure location.
If you are using Vault for secrets management, Lambda requires access to Vault for authentication and to retrieve secrets.
Including the Vault Lambda Extension in the execution environment enables functions to retrieve secrets from Vault.
With the Vault Lambda Extension with your Lambda runtime, and setting the proper environment variables, the function will be able to retrieve secrets from Vault before the function executes. The function now has the credentials needed to implement your business logic.
Lambda creates instances of serverless functions as needed, which the structure includes a few things like a runtime, environment variables and the executable/interpreter. Lambda has standard runtimes for many languages available, but if that runtime does not include a particular library or dependency it can be added a few ways. One of particular interest is the Lambda layer, which is an addition of a library, module or dependency to the standard Lambda runtime for your language. When a Lambda function is invoked, these are all loaded into the runtime process and then executed.
Extensions are additions to the runtime process and can added as a Lambda layer to your runtime environment. If the proper environment variables are set, the Vault AWS Lambda Extension will run authenticate and securely retrieve secrets before your Lambda function invokes.
First, using a Terraform config you will create the function with supporting IAM Roles and Policies, and review the infrastructure that is created.
You will then set up HCP Vault Dedicated to allow access from your AWS Account to Vault Dedicated via an IAM Role. After the initial Vault Dedicated setup, you will create a Vault Role and Policy that allow the execution role the Lambda function uses to access one KV Secret (kv/data/test/lambda) in your instance of Vault Dedicated.
Next, you will create a secret in Vault, and then set up AWS Authentication to enable the function you created earlier to connect to your Vault Dedicated instance.
Finally, you will observe it all working together.
This tutorial requires an AWS account, Terraform, Vault CLI and the example Terraform configuration to create a demonstration environment.
VAULT_TOKEN and VAULT_ADDR.Clone the Terraform configuration and Lambda function code from the following repository:
$ git clone https://github.com/hashicorp-education/learn-vault-intro-lambda-extension
Change into the repository directory.
$ cd learn-vault-intro-lambda-extension
The repository contains Terraform configuration to create the following resources:
In HCP, under Access control (IAM) choose Service Principals.
From the Service Principals page, choose the link that your service principle's name uses and go into the details page.
in the Create service principal key page, choose + Generate key.
Copy the Client ID then, in a terminal, set the HCP_CLIENT_ID environment variable to the copied value.
$ export HCP_CLIENT_ID=<COPIED_CLIENT_ID>
Switch back to the HCP Portal and copy the Client Secret then, in a terminal, set the HCP_CLIENT_SECRET environment variable to the copied value.
$ export HCP_CLIENT_SECRET=<COPIED_CLIENT_SECRET>
Terraform is now able to authenticate with HCP.
Set AWS_ACCESS_KEY_ID to store your AWS Access Key.
$ export AWS_ACCESS_KEY_ID=<<AWS Access Key here>>
Set AWS_SECRET_ACCESS_KEY to store your AWS Secret Access Key.
$ export AWS_SECRET_ACCESS_KEY=<<AWS Secret Access Key here>>
Now set the target AWS region.
$ export TF_VAR_aws_region=us-west-2
Run terraform init to initialize the Terraform configuration.
$ terraform init
...snip...
Terraform has been successfully initialized!
...snip...
Package the Lambda function through the build script.
$ ./build.sh
$PWD/learn-vault-intro-lambda-extension/demo-function $PWD/learn-vault-intro-lambda-extension
Removing old builds...
Making new zip...
adding: handler.py (deflated 31%)
$PWD/learn-vault-intro-lambda-extension
Apply the changes.
$ terraform apply
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
The terminal output displays the plan that it found and the resources it creates.
Enter yes to confirm and resume.
Note
Keep in mind that answering yes at this time creates actual resources with associated costs.
Verify that when the terraform apply command completes, smd you see the following.
...snip...
Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
Take a quick tour to learn what was provisioned by Terraform in the AWS and HCP UI.
Launch your AWS Account Console and open the Lambda console. Select the vault-lambda-extension-demo-function function.
Scroll down to the Code Source tab, and examine the Python code.
import json
import os
def lambda_handler(event, context):
# the location of secret
secret_file = os.environ['VAULT_SECRET_FILE_DB']
f = open(secret_file, "r")
# put the json object into a python dictionary
dct = json.loads(f.read())
# print off your API Key
print("API Key: ", dct["data"]["data"]["api-key"])
This code is straightforward, getting the location of the secret that is returned from Vault from an environment variable, reading it into a Python map and then printing it to standard output. Understanding the environment variables are critical to understanding the Vault lambda extension.
Click on the Configuration tab and click on the environment variables section on the left.
The environment variables should be similar to as follows.
The Vault Lambda Extension is configured via these environment variables. So VAULT_AUTH_PROVIDER indicates it should use AWS Authentication, VAULT_NAMESPACE indicates the namespace that will be used, and so forth. Please see the previous link for details.
In order to use the Vault CLI locally, you need to go to the HCP UI, find your cluster to get the Vault token, and address.
Choose your Vault instance and click on Access Vault, and choose Command-line (CLI).
From there some variables need to set in your terminal.
$ export VAULT_ADDR="[YOUR_CLUSTER_ADDER_HERE]"; export VAULT_NAMESPACE="admin"
You still need to get the token. Go back to the main screen for your Vault cluster and find for the New admin token and choose Generate token. Generating it places it in your paste buffer. Copy the below and then set the admin token with:
$ export VAULT_TOKEN=[YOUR_TOKEN_HERE]
Note
This token expires in 6 hours.
For more details. please refer to the configure development hosts, paying attention to the HCP sections for details on setting environment variables.
Create a standard KV secret through CLI.
Check Vault status to make sure the CLI is properly set up.
$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 1
Threshold 1
Version 1.12.0+ent
Build Date 2022-10-10T19:00:46Z
Storage Type raft
Cluster Name vault-cluster-2127cf0f
Cluster ID 28bcf54f-94ec-2eed-1e54-2d67b58bbaaf
HA Enabled true
HA Cluster https://XXX.XX.XX.XXX:8201
HA Mode active
Active Since 2022-10-26T16:57:12.105415484Z
Raft Committed Index 413
Raft Applied Index 413
Last WAL 173
Enable KV Secrets Engine v2.
$ vault secrets enable -version=2 kv
Success! Enabled the kv secrets engine at: kv/
Create a secret named api-key.
$ vault kv put kv/test/lambda api-key="ABCDEFG9876"
=== Secret Path ===
kv/data/test/lambda
======= Metadata =======
Key Value
--- -----
created_time 2022-10-27T16:30:34.405574019Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
In order for the Lambda function to be able to contact the Vault Dedicated instance you need AWS authentication set up.
List the authentication is already enabled.
$ vault auth list
Path Type Accessor Description
---- ---- -------- -----------
token/ ns_token auth_ns_token_dd101c2c token based credentials
Now create a policy called vault-policy-for-aws-lambda-role, which allows read access the secret created above.
$ vault policy write vault-policy-for-aws-lambda-role - << EOF
# Grant 'read' permission to paths prefixed by 'kv/data/test/lambda'
path "kv/data/test/lambda" {
capabilities = [ "read" ]
}
EOF
List the available policies.
$ vault policy list
default
hcp-root
vault-policy-for-aws-lambda-role
For your lambda function to be able to connect to your instance of Vault Dedicated, go ahead and enable AWS Authentication.
$ vault auth enable aws
Success! Enabled aws auth method at: aws/
List the available auth methods.
$ vault auth list
Path Type Accessor Description
---- ---- -------- -----------
aws/ aws auth_aws_b3ac2184 n/a
token/ ns_token auth_ns_token_dd101c2c token based credentials
Configure AWS Authentication with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY already set up.
$ vault write auth/aws/config/client secret_key=$AWS_SECRET_ACCESS_KEY access_key=$AWS_ACCESS_KEY_ID
Success! Data written to: auth/aws/config/client
Create an AWS auth method role in Vault that gives access to the IAM role vault-lambda-extension-demo-lambda-role and attach the policy vault-policy-for-aws-lambda-role to it.
Note
Update the <YOUR_ACCOUNT_ID> with your AWS Account ID in the following command before running it.
$ vault write auth/aws/role/vault-role-for-aws-lambdarole auth_type=iam bound_iam_principal_arn=arn:aws:iam::<YOUR_ACCOUNT_ID>:role/vault-lambda-extension-demo-lambda-role policies=vault-policy-for-aws-lambda-role
Success! Data written to: auth/aws/role/vault-role-for-aws-lambdarole
List the roles.
$ vault list /auth/aws/role
Keys
----
vault-role-for-aws-lambdarole
Examine the details of vault-role-for-aws-lambdarole.
$ vault read auth/aws/role/vault-role-for-aws-lambdarole
Key Value
--- -----
allow_instance_migration false
auth_type iam
bound_account_id []
bound_ami_id []
bound_ec2_instance_id <nil>
bound_iam_instance_profile_arn []
bound_iam_principal_arn [arn:aws:iam::144830530561:role/vault-lambda-extension-demo-lambda-role]
bound_iam_principal_id [AROA3SCJYFFCPXMX5QMOP]
bound_iam_role_arn []
bound_region []
bound_subnet_id []
bound_vpc_id []
disallow_reauthentication false
inferred_aws_region n/a
inferred_entity_type n/a
policies [vault-policy-for-aws-lambda-role]
resolve_aws_unique_ids true
role_id b821fe4b-ff5c-22ac-9594-7f95858a64fc
role_tag n/a
token_bound_cidrs []
token_explicit_max_ttl 0s
token_max_ttl 0s
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [vault-policy-for-aws-lambda-role]
token_ttl 0s
token_type default
Go back to the AWS Console, find vault-lambda-extension-demo-function and choose the Test tab and press Test button.
Examine the output. In addition to a summary of execution, find for text similar to the below.
2022-10-28T14:45:04.456Z [INFO] vault-lambda-extension: Initialising
2022-10-28T14:45:04.667Z [INFO] vault-lambda-extension: Initialised
2022-10-28T14:45:04.667Z [INFO] vault-lambda-extension: Starting HTTP proxy server
220140c1-4d3c-4656-b9ba-a6df57a1f1d82022-10-28T14:45:04.668Z [INFO] vault-lambda-extension: Waiting for event...
EXTENSION Name: vault-lambda-extension State: Ready Events: [INVOKE,SHUTDOWN]
START RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5 Version: $LATEST
2022-10-28T14:45:04.774Z [INFO] vault-lambda-extension: Received event
2022-10-28T14:45:04.774Z [INFO] vault-lambda-extension: Waiting for event...
API Key: ABCDEFG9876
END RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5
REPORT RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5 Duration: 1.54 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 65 MB Init Duration: 437.45 ms
From the line with API Key: ABCDEFG9876 you know that the extension retrieved the API_KEY value from Vault.
Clean up your infrastructure provisioned by Terraform.
The terraform destroy deletes the HCP Cluster, Lambda function and all the support structures - IAM Role, Policy, etc.
Delete the Terraform state.
In the AWS IAM Console, find the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID you created for this tutorial and make them inactive and delete them both.
Unset all the environment variables used in this tutorial.
$ unset AWS_SECRET_ACCESS_KEY; unset AWS_ACCESS_KEY_ID; unset HCP_CLIENT_ID; unset HCP_CLIENT_SECRET
Check the environment variables were unset:
If the variables were successfully unset they will not appear in the result of this command. An execution and successful unset will provide no output.
In this tutorial you built and deployed a Lambda function using the Vault AWS Lambda Extension using Terraform. If you are interested in more detail on using Terraform to manage Vault Dedicated refer to Manage Codified Vault on HCP Vault Dedicated with Terraform That simple function retrieved a simple KV secret from Vault, and printed it to the CloudWatch logs. The extension is configured by the Lambda environment variables, and automatically authenticates and retrieves a secret before the function is invoked.
You can learn more about the Vault AWS Lambda Extension by reading the blog post Use AWS Lambda Extensions to Securely Retrieve Secrets From HashiCorp Vault and by reviewing the Vault AWS Lambda Extension code repository. Of particular interest are the environment variables used to configure the extension.
Next, you enabled AWS Authentication for Vault Dedicated, and you can learn more through the Set up AWS Auth Method for HCP Vault Dedicated tutorial and the documentation AWS Authentication.
If you are interested in a lab with a more complex example, you can try the Lambda Vault Extension tutorial. That tutorial has a more detailed scenario using a secret to then access a MYSQL Database. In addition there is a section on caching the secret retrieved from Vault.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4