AWS ECS Deploy Blue/Green using GitHub Actions.
Sorry if this didn't work. I tried to do this all in ONE go, or one commit. ~ Andrew C. 29 May 2020. Ripperoni.
Edit: It works! It only took so long... and 3 days of finally realizing it security groups is a thing... I'm a beginner, so start-ups, take advantage of this repository! ~ Andrew C. 10 June 2020.
Much appreciated code taken from awslabs: https://github.com/awslabs/ecs-nginx-reverse-proxy/tree/master/reverse-proxy
License: Apache Software License 2.0Table of contents:
The deployment is done through Travis CI which does a sorta webhook to a GitHub action which will use several AWS GH Actions to finally deploy your application. TODO Actually create that webhook.
The AWS services that we'll be using are: CodeDeploy, ECS + ECR, Parameter Store, IAM, Application Load Balancer. CodeDeploy is for deploying from the GH action. ECS + ECR is where our servers will be located. Parameter store will be where we store our secrets/environment variables. IAM is for proper security measures of the credentials given to the server AND to GitHub for deployment. Yes, we need two IAM users, one for GH and the other for ECS. Finally, we need ALB for proper Blue/Green deployment assuming you have more than one instance in your ECS cluster.
I chose ECS because there were GitHub actions for this anyways. However, note that this is much slower than simply using EC2. The current Travis configuration uses Docker, but my PR in cookiecutter-django should change the configuration from Docker based to completely Docker free, erasing 1.5 minutes of CI.
Why not use Fargate? Personally, I like EC2 over Fargate or Lambda since I use celery a lot. That's really it; I haven't used AWS much, nor have I actually deployed an AWS app until now.
I've set up an nginx reverse proxy to do a lot of heavy weighted work and to minimize security risks. Additionally, I've left out several key components like PostgreSQL and Redis. Those can easily be added in the aws-task-definition.json. Just look at how the rest of the GH action is setup to configure PostgreSQL Dockerfile in compose/production/postgres.
Let's start the deployment process!
The following details how to deploy this application stack to ECS. If you need to also setup your database and cache, follow [issue #9](#9). You may want to get the hang of the following first, specifically security groups in step 1. Also note that I didn't write that in the tutorial because it's spare and came from my memory from a couple months back. I didn't think it was detailed enough for this tutorial.
It's a lot of instructions since there are so many services:
1. You must have an IAM user with the correct permissions that you can find at the bottom of this README. Make sure you copy the ones from the section labeled Minimal IAM Credentials for Deployment. You can create a new policy using the JSON below during the Set Permissions section by pressing "Attach existing policies directly" and pressing "Create Policy."
Add the IAM user's credentials to your repo's secrets in the repo's settings. The credentials' names MUST be AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
It took me 10 prolonged days to figure out my security groups were not properly configured for my ALB and ECS instances. So follow this carefully.
Create a security group called WebAccess with a description saying it's for ALB.
The inbound traffic should be of 4 rules:
The outbound rules should be left on default, but just in case:
Create another security group. This is for your ECS instances. Name it ECS-reverse-proxy (for another website, I'd recommend prefixing the name of this with your website name).
The inbound traffic should be 1 rule only:
The outbound traffic is the same as the first one.
aws ecr create-repository --repository-name nginx-reverse-proxy --region us-east-2 aws ecr create-repository --repository-name django-app --region us-east-2
Replace the region with your AWS region, and make sure you change it in the .github/workflows/aws.yml file, as well.
4. Create an ECS cluster. Replace the values for cluster in the aws workflow with your cluster name. The default is "cookiecutter-django". I guess you could just write that one and not need to change the one in the GH action.
In order to add our environment variables via our task definition, we must make sure the IAM role (above, hopefully it was ecsInstanceRole) can even do a task execution.
Go to your newly created IAM role and click "Trust relationships"
Edit the trust relationship so that, in the "Service" array, you add ecs-tasks.amazonaws.com
I bought a random website with a .de ending since that came out to be $8. My website was asdfasq.de. Random, ey?
The more random the name and extension, the cheaper.
Change allowed hosts in config/settings/production.py to your domain.
Change every instance of asdfasq.de in compose/production/ecs/nginx/nginx.conf to your domain.
Find ACM (certificate manager) and add your domain and its www. format, as well.
NOTE: I might be missing a step with the certificate manager. I deployed a test website on EC2 as a standalone, and I might've done something to properly configure the certificate. PLEASE open a PR/Patch if I'm missing it.
Go to the EC2 page. Find the Load Balancers section and create a new balancer.
Go back to Route 53. Go to your hosted zone and add 2 A record sets. Choose yes for use alias. Find your load balancer.
The difference between each record set is that the first one for name can be left blank while the other one should have www. This is also how you can have multiple ECS clusters for different applications (i.e. with subdomains).
Go to the aws-task-definition.json file and copy its contents.
In the ECS dashboard, create a new task definition. Scroll to the bottom until you find "configure via JSON." Paste the contents.
After you finished creating your cluster, you should arrive in the service tab. Create a service.
While you're creating the service, the review stage should show your new target groups. If not, it's fine. The task will stop and regenerate.
Right click on each target group and change the success codes at the bottom from 200 to 200,301 (you cannot add spaces).
Search up Systems Manager. Look for Parameter Store on the left side. You'll need to add the parameters from .envs/.production/template.django.
I've noted which ones you should add.
If you tested this first on a random GitHub repository, here's how to clean those resources up:
I didn't want to make ANOTHER image just for Celery; instead, I just used:
>> celery multi start -A config.celery_app worker beat
I use Sentry to log all my Celery stuff, anyways, and it will come with cookiecutter-django if you opt-in.
I also use RDS for PostgreSQL and ElastiCache for Redis. You don't HAVE to, but that would mean you need to configure some more stuff in the aws-task-definitions.json.
In the task definition, you can easily add the redis and PostgreSQL images. If you follow the GitHub action of how I set up everything and how you can easily use the Dockerfile in compose/production/postgres, then just follow how I did the Django app.
Initial Cookiecutter Generationproject_name [My Awesome Project]: AWS ECS Deploy project_slug [aws_ecs_deploy]: description [Behold My Awesome Project!]: AWS ECS Deploy (hopefully with Blue/Green) using GitHub Actions author_name [Daniel Roy Greenfeld]: Andrew Chen Wang domain_name [example.com]: email [andrew-chen-wang@example.com]: acwangpython@gmail.com version [0.1.0]: Select open_source_license: 1 - MIT 2 - BSD 3 - GPLv3 4 - Apache Software License 2.0 5 - Not open source Choose from 1, 2, 3, 4, 5 [1]: 4 timezone [UTC]: windows [n]: use_pycharm [n]: use_docker [n]: y Select postgresql_version: 1 - 11.3 2 - 10.8 3 - 9.6 4 - 9.5 5 - 9.4 Choose from 1, 2, 3, 4, 5 [1]: Select js_task_runner: 1 - None 2 - Gulp Choose from 1, 2 [1]: Select cloud_provider: 1 - AWS 2 - GCP 3 - None Choose from 1, 2, 3 [1]: Select mail_service: 1 - Mailgun 2 - Amazon SES 3 - Mailjet 4 - Mandrill 5 - Postmark 6 - Sendgrid 7 - SendinBlue 8 - SparkPost 9 - Other SMTP Choose from 1, 2, 3, 4, 5, 6, 7, 8, 9 [1]: 2 use_drf [n]: custom_bootstrap_compilation [n]: use_compressor [n]: use_celery [n]: y use_mailhog [n]: use_sentry [n]: use_whitenoise [n]: use_heroku [n]: Select ci_tool: 1 - None 2 - Travis 3 - Gitlab Choose from 1, 2, 3 [1]: keep_local_envs_in_vcs [y]: debug [n]:Minimal IAM Credentials for ECS
You'll need these permissions for your ECS: - S3 Full Access
Minimal IAM Credentials for DeploymentYou're probably thinking... wtf is with all these brackets. Security. Besides that, you can use asterisks for demonstration for demonstration purposes.
For me, during testing, I just used FullAccess... Shh...
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"RegisterTaskDefinition",
"Effect":"Allow",
"Action":[
"ecs:RegisterTaskDefinition"
],
"Resource":"*"
},
{
"Sid":"PassRolesInTaskDefinition",
"Effect":"Allow",
"Action":[
"iam:PassRole"
],
"Resource":[
"arn:aws:iam::<aws_account_id>:role/<task_definition_task_role_name>",
"arn:aws:iam::<aws_account_id>:role/<task_definition_task_execution_role_name>"
]
},
{
"Sid":"DeployService",
"Effect":"Allow",
"Action":[
"ecs:DescribeServices",
"ecs:UpdateService",
"codedeploy:GetDeploymentGroup",
"codedeploy:CreateDeployment",
"codedeploy:GetDeployment",
"codedeploy:GetDeploymentConfig",
"codedeploy:RegisterApplicationRevision"
],
"Resource":[
"arn:aws:ecs:<region>:<aws_account_id>:service/<cluster_name>/<service_name>",
"arn:aws:codedeploy:<region>:<aws_account_id>:deploymentgroup:<application_name>/<deployment_group_name>",
"arn:aws:codedeploy:<region>:<aws_account_id>:deploymentconfig:*",
"arn:aws:codedeploy:<region>:<aws_account_id>:application:<application_name>"
]
}
]
}
How do I add celery?
Go to compose/production/ecs/django/start and add the line
celery multi start worker beat -A config.celery_app
If you'd like to troubleshoot your AWS actions, add the secret ACTION_STEP_DEBUG with value true to your GitHub repo.
Here is the AWS action doc specifying this https://github.com/aws-actions/amazon-ecs-deploy-task-definition#troubleshooting
What's this license?
Apache 2.0
Best practices?
Rotate your keys!
What if I mess up creating the ECS service?
Got something there's a service already here? I did too, lol. Search up AWS Cloud Map. Delete the one that says local.
You may also have to go to CodeDeploy and delete the Application there, too.
Are you experienced in AWS?
Absolutely not. This would be my first time actually using AWS besides self hosting on one instace. This was just a nice learning experience that seems sooooo painful for start ups. In other words, STARTUPS! Get moving! I just gave you a free repo to copy off of :)
I did play around with AWS trying to use the default cookiecutter-django before which is why I didn't know how I set up ACM in the first place. It worked after a painful 12 hours of trying to figure out wtf was going wrong.
Why do you like typing so much?
I like to train my fingers.
Plus, it's nice seeing my painful moments and learning from them. It's like the cliche standing back and being proud of your work.
But this was a painful 10 hours... I started at 12 and now it's 22:11.
What did you learn from this?
Always start small. On 10 June 2020, I finally figured to try and start small with a single EC2 with a load balancer (however, I will admit that I suspected the security groups was an issue for the most part).
On the same day, I finally got it to work. So, always start small, and then try out this methodology.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4