The Terraform state file is the source of truth for your infrastructure. Protecting and backing up this file is critical for practitioners who use Terraform in production. Remote state storage with HCP Terraform offers fail-safes for your infrastructure in the event of disaster-recovery situations and local file corruption. Although Terraform takes steps to prevent state errors, your state file can get corrupted due to partial apply operations or incorrectly running terraform import or terraform taint. Using the HCP Terraform API, you can safely download, modify, and upload your state file to an HCP Terraform workspace.
In this tutorial, you will generate a state file by deploying an AWS instance with web access using the Terraform CLI. Then, you will download your remote state file and use the Terraform API to create a new state version.
For this tutorial you will need:
awscli configured.jq installed.Clone the example GitHub repository.
$ git clone https://github.com/hashicorp-education/learn-tfc-state-api
Change into the repository directory.
Open the main.tf file to review the configuration. The main resources in this configuration are an AWS EC2 instance and a security group with port 8080 access.
Next, open the terraform.tf file to configure your HCP Terraform organization. Replace <ORGANIZATION-NAME> with your HCP Terraform organization name.
terraform.tf
terraform {
required_version = "~> 1.10"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.82.2"
}
}
cloud {
organization = "<ORGANIZATION_NAME>"
workspaces {
name = "learn-terraform-state-api"
}
}
}
Save your changes.
Run the terraform login subcommand, and follow the prompts to authenticate to HCP Terraform.
$ terraform login
Terraform will request an API token for app.terraform.io using your browser.
If login is successful, Terraform will store the token in plain text in
the following file for use by subsequent commands:
/Users/<YOU>/.terraform.d/credentials.tfrc.json
Do you want to proceed?
Only 'yes' will be accepted to confirm.
Enter a value: yes
---------------------------------------------------------------------------------
Terraform must now open a web browser to the tokens page for app.terraform.io.
If a browser does not open this automatically, open the following URL to proceed:
https://app.terraform.io/app/settings/tokens?source=terraform-login
---------------------------------------------------------------------------------
Generate a token using your browser, and copy-paste it into this prompt.
Terraform will store the token in plain text in the following file
for use by subsequent commands:
/Users/<YOU>/.terraform.d/credentials.tfrc.json
Token for app.terraform.io:
Enter a value:
Retrieved token for user <YOU>
---------------------------------------------------------------------------------
-
----- -
--------- --
--------- - -----
--------- ------ -------
------- --------- ----------
---- ---------- ----------
-- ---------- ----------
Welcome to HCP Terraform! - ---------- -------
--- ----- ---
Documentation: terraform.io/docs/cloud -------- -
----------
----------
---------
-----
-
New to HCP Terraform? Follow these steps to instantly apply an example configuration:
$ git clone https://github.com/hashicorp/tfc-getting-started.git
$ cd tfc-getting-started
$ scripts/setup.sh
For more detailed instructions on logging in, reference the Authenticate the CLI with HCP Terraform tutorial.
After authenticating, initialize your Terraform configuration.
$ terraform init
Initializing HCP Terraform...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v5.82.2...
- Installed hashicorp/aws v5.82.2 (signed by HashiCorp)
HCP Terraform has been successfully initialized!
You may now begin working with HCP Terraform. Try running "terraform plan" to
see any changes that are required for your infrastructure.
If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.
Navigate to HCP Terraform and select your learn-terraform-state-api workspace. Click on "Variables" and add your region variable as a Terraform variable. Add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables as environment variables. Be sure to mark the AWS credentials environment variables as sensitive
Note
The AWS_SESSION_TOKEN is optional unless your organization requires it.
In your terminal, apply your configuration. Enter yes when prompted to confirm your changes.
$ terraform apply
Running apply in HCP Terraform. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.
Preparing the remote apply...
To view this run in a browser, visit:
https://app.terraform.io/app/YOUR-ORG/learn-terraform-state-api/runs/run-Lm96BJVNXkRv7dNQ
Waiting for the plan to start...
##...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Outputs:
instance_id = "i-0bcc0850e611cfd6b"
public_ip = "3.143.170.167"
Verify that your state file contains your resources using terraform state list command.
$ terraform state list
data.aws_ami.ubuntu
aws_instance.example
aws_security_group.sg_web
aws_security_group_rule.sg_web
In the next section, you will configure your HCP Terraform workspace.
You will need your workspace ID and a new HCP Terraform API token to access your workspace's state file.
Retrieve workspace IDIn HCP Terraform, navigate to your new workspace.
In your learn-terraform-state-api workspace, navigate to "Settings" then "General" and copy your workspace ID.
Create a workspace ID environment variable in your terminal, replacing <YOUR-WORKSPACE-ID> with the workspace ID you just copied.
$ export WORKSPACE_ID=<YOUR-WORKSPACE-ID>
Create a new API token by clicking the icon for your user in the top left corner, then Account Settings. Then select Tokens from the nav menu. Click the Create an API token button to create a new token.
Name your token "learn-terraform-state-api" then select "Create API token".
Copy the API token.
Create an environment variable with your token, replacing <YOUR-TFC-TOKEN> with the token you just copied.
$ export TFC_TOKEN=<YOUR-TFC-TOKEN>
Click on the lock icon to lock your workspace. Locking your workspace prevents other operations from running and potentially corrupting the state file you are going to download. Your workspace needs to be locked before you can push a new state file via API. You must lock the workspace as the same user you generated the HCP Terraform token for in the previous step.
Run the AWS CLI to add the Org tag to your EC2 resource. Your state file does not have a record of this value.
$ aws ec2 create-tags --region $(terraform output -raw region) --resources $(terraform output -raw instance_id) --tags Key=Org,Value=HashiCorp
Your new Org tag is set to HashiCorp in AWS, but your Terraform state file does not reflect this change.
Later in this tutorial, you will reconcile this difference with the Terraform state.
In your terminal, navigate to the helper_scripts folder.
The shell scripts in this directory construct your API queries, download your remote state file for editing, and create a payload for uploading your changes.
Open the getstate.sh file to review the API query.
getstate.sh
#!/bin/bash
HTTP_RESPONSE=$(curl \
--header "Authorization: Bearer "$TFC_TOKEN"" \
--header "Content-Type: application/vnd.api+json" \
"https://app.terraform.io/api/v2/workspaces/"$WORKSPACE_ID"/current-state-version" | jq -r '.data | .attributes | ."hosted-state-download-url"')
curl -o state.tfstate $HTTP_RESPONSE
If you are using Terraform Enterprise, change the URL from app.terraform.io to your personalized Terraform Enterprise domain.
This snippet uses your environment variables and authenticates to your HCP Terraform workspace to download the current remote state file. The hosted-state-download-url contains the URL that hosts your remote state file.
In your terminal, run the getstate.sh script.
$ ./getstate.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1995 100 1995 0 0 8711 0 --:--:-- --:--:-- --:--:-- 8711
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8941 0 8941 0 0 33486 0 --:--:-- --:--:-- --:--:-- 33486
Verify your query downloaded the state file and open state.tfstate.
$ cat state.tfstate
{
"version": 4,
"terraform_version": "1.10.3",
"serial": 0,
"lineage": "b9b6c592-742c-c303-d1f7-8b2f8c935feb",
##...
Warning
We discourage directly editing state files. In production environments, you should only use this method as a last resort.
Now that you have the remote state downloaded, open the state.tfstate file in your file editor. Increment your serial number by one then save the file.
state.tfstate
{
"version": 4,
"terraform_version": "1.10.3",
- "serial": 0,
+ "serial": 1,
"lineage": "b9b6c592-742c-c303-d1f7-8b2f8c935feb",
"outputs": {
"instance_id": {
"value": "i-01b237003b468184d",
"type": "string"
},
"public_ip": {
"value": "3.19.246.180",
"type": "string"
}
},
## ...
}
This is your new state version number. Terraform uses the serial to keep track of the changes made in each new state file and uses it to make sure your operations run against the correct known state file in the HCP Terraform workspace. In standard operations, Terraform updates the serial for you automatically. However, since you're pushing a new state version, you need to manually increment this value.
Edit your instance tags to include the new Org tag. Search for terraform-learn-state-versioning in the state.tfstate file. Add a comma at the end of the Name tag. Terraform parses the JSON-formatting state file and adds elements to your resource records.
state.tfstate
##...
"tags": {
- "Name": "terraform-learn-state-versioning"
+ "Name": "terraform-learn-state-versioning",
+ "Org": "HashiCorp"
},
"tags_all": {
- "Name": "terraform-learn-state-versioning"
+ "Name": "terraform-learn-state-versioning",
+ "Org": "HashiCorp"
},
"tenancy": "default",
##...
Save your file.
Now, you will construct your current state payload. Select the tab for your operating system for specific instructions.
In your helper_scripts folder, open the createpayload.sh file.
helper_scripts/createpayload.sh
#!/bin/bash
serial=$(cat state.tfstate | jq '.serial')
md5_compute=$(md5 -q state.tfstate)
md5=\"$md5_compute\"
lineage=$(cat state.tfstate | jq '.lineage')
base64_encode=$(base64 state.tfstate)
state=\"$base64_encode\"
echo "{
\"data\": {
\"type\": \"state-versions\",
\"attributes\": {
\"serial\": $serial,
\"md5\": "$md5",
\"lineage\": "$lineage",
\"state\": "$state"
}
}
}" > payload.json
This snippet finds the serial and lineage values in your state.tfstate file and creates an MD5 signature for your state file with a base64 encoded version of your state. Then, this script passes those values to a new file named payload.json. You will upload this file to your HCP Terraform workspace with the TFC API in the next step.
Run the createpayload.sh script.
Open payload.json to verify the script successfully created your payload.
helper_scripts/payload.json
$ cat payload.json
{
"data": {
"type": "state-versions",
"attributes": {
"serial": 1,
"md5": "f51e44f5672b40725e283c1bd5556752",
"lineage": "939c75bf-0872-6277-d273-3df86f7ac679",
"state": "ewogICJ2ZXJzaW9uIjogNCwKICAidGVyc
## …
}
In your helper_scripts folder, open the linux-createpayload.sh snippet.
helper_scripts/linux-createpayload.sh
#!/bin/bash
serial=$(cat state.tfstate | jq '.serial')
md5_compute=$(md5sum state.tfstate | awk '{print $1}')
md5=\"$md5_compute\"
lineage=$(cat state.tfstate | jq '.lineage')
base64_encode=$(base64 state.tfstate)
state=\"$base64_encode\"
echo "{
\"data\": {
\"type\": \"state-versions\",
\"attributes\": {
\"serial\": $serial,
\"md5\": "$md5",
\"lineage\": "$lineage",
\"state\": "$state"
}
}
}" > payload.json
This snippet finds the serial and lineage values in your state.tfstate file and creates an md5 signature for your state file with a base64 encoded version of your state. Then, this script passes those values to a new file named payload.json. You will upload this file to your HCP Terraform workspace with the TFC API in the next step.
Run the linux-createpayload.sh script.
$ ./linux-createpayload.sh
Verify the script created your payload.json file correctly.
$ cat payload.json
{
"data": {
"type": "state-versions",
"attributes": {
"serial": 1,
"md5": "f51e44f5672b40725e283c1bd5556752",
"lineage": "939c75bf-0872-6277-d273-3df86f7ac679",
"state": "ewogICJ2ZXJzaW9uIjogNCwKICAidGVyc
## ...
}
Now that you have a JSON payload with your encrypted state file, upload the new state file to HCP Terraform.
In your helper_scripts directory, open the uploadstate.sh file.
helper_scripts/uploadstate.sh
#!/bin/bash
HTTP_POST=$(curl \
--header "Authorization: Bearer "$TFC_TOKEN"" \
--header "Content-Type: application/vnd.api+json" \
--request POST \
--data @payload.json \
"https://app.terraform.io/api/v2/workspaces/"$WORKSPACE_ID"/state-versions")
echo $HTTP_POST
Tip
If you are using Terraform Enterprise, change the URL from app.terraform.io to your Terraform Enterprise domain.
This API query uses the --data flag to upload the payload.json file to your workspace.
Run the uploadstate.sh script.
$ ./uploadstate.sh
##...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13247 100 1107 100 12140 1392 15270 --:--:-- --:--:-- --:--:-- 16641
{"data":{"id":"sv-VBU3yeG5XMLgK5K6","type":"state-versions","attributes":{"created-at":"2021-04-08T21:59:29.954Z","size":null,"hosted-state-download-url":"https://archivist.terraform.io/v1/object/dmF1bHQ6djE6ZXFONmlaYlVhcHVNOE9WWENZZkljdmJz..."
##...
In your HCP Terraform workspace, navigate to your "States" tab and select the most recent state.
In the "Changes in this version" section, confirm your new state file contains a new serial number and tag reference.
Unlock your workspace by clicking on the lock icon and confirming the unlock.
Because you updated your resource outside of the Terraform workflow with an additional tag, you must update the configuration with the updated resource and run a terraform apply to maintain parity with your state file. Terraform will perform the apply, but will not make any resource changes. This is a "no-operation" or "no-op" apply.
Change into the root directory.
Open the main.tf file and update your instance tag.
main.tf
resource "aws_instance" "example" {
ami = data.aws_ami.ubuntu.id
instance_type = "t2.micro"
vpc_security_group_ids = [aws_security_group.sg_web.id]
user_data = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y apache2
sed -i -e 's/80/8080/' /etc/apache2/ports.conf
echo "Hello World" > /var/www/html/index.html
systemctl restart apache2
EOF
tags = {
Name = "terraform-learn-state-versioning"
+ Org = "HashiCorp"
}
}
Run terraform apply to consolidate your configuration with your remote state. This is a "no-op" apply.
$ terraform apply
Running apply in HCP Terraform. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.
Preparing the remote apply...
To view this run in a browser, visit:
https://app.terraform.io/app/<YOUR-ORG>/learn-terraform-state-api/runs/run-gGFN9Tdd6cGuaqyN
Terraform v0.15.1
on linux_amd64
Configuring remote state backend...
Initializing Terraform configuration...
aws_security_group.sg_web: Refreshing state... [id=sg-08df7f3f965e47a6a]
aws_security_group_rule.sg_web: Refreshing state... [id=sgrule-4136193275]
aws_instance.example: Refreshing state... [id=i-0a8f43386c25bc073]
No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and the remote system(s). As a result, there are no actions to
take.
Now that your modified state file matches your infrastructure and configuration, delete the local version of your state file.
$ rm helper_scripts/state.tfstate
Your payload.json file also contains an encrypted version of your state. Delete your payload.json file.
$ rm helper_scripts/payload.json
In HCP Terraform, destroy your remote workspace.
Navigate to "Settings" > "Destruction and Deletion".
At the bottom of the page, select "Queue Destroy Plan" and confirm.
Next, delete your workspace from HCP Terraform. At the bottom of the page, select "Delete workspace" and confirm.
In this tutorial, you learned how to use the HCP Terraform API to interact with and update your HCP Terraform workspace's state. First, you created infrastructure in HCP Terraform. Then, you downloaded your current HCP Terraform state file and safely modified and versioned your state file. Finally, you uploaded your versioned and updated state file to HCP Terraform.
For more information about the HCP Terraform API or Terraform state, review the following resources:
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4