AWS/CLI
Install awscli
(command line)
curl -O https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install awscli
or
sudo apt-get install awscli #it will update a lot of packages to Python3 but will leave 2.7 as default
Configure AWS cli credentials
- Environment variables
AWS_ACCESS_KEY_ID=1111 AWS_SECRET_ACCESS_KEY=2222 AWS_DEFAULT_REGION=eu-west-1
- CLI credentials file
When you run aws configure
and enter credentials, they are stored in a file at ~/.aws/credentials. Additionally, some configuration settings—such as the default region—are stored at ~/.aws/config.
- ~/.aws/credentials vs ~/.aws/config
config
- contains config, profile name is prefixed with word profile, eg:[profile <profile-name>]
credentials
- contains credentials for a profile, the profile name is not prefixed, eg:[<profile-name>]
. Note credentials can be stored in the config file as well.
$ aws configure #sets up default profile $ aws configure --profile dev #configure named 'dev' profile $ aws configure --profile piotr #configure named 'piotr' profile $ aws ec2 describe-regions #to get a list all available regions
The /.aws/config profile sections must have the format of [profile profile-name], except for the default profile. For example:
Example ~/.aws/credentials file
[default] aws_access_key_id=111xxxxxxxxxxxxxxxxx aws_secret_access_key=111xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [profile dev] aws_access_key_id=222xxxxxxxxxxxxxxxxx aws_secret_access_key=222xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [profile piotr] aws_access_key_id=333xxxxxxxxxxxxxxxxx aws_secret_access_key=333xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
You can also use environment variables described here
Configure AWS cli with Cross-Account Roles & MFA
Let’s assume that we have 4 AWS accounts:
- One identity-account (acc_no: 000xxxxxxxxx) where we provision IAM users, create their API Credentials and assign MFA -
- user 'piotr' mfa_arn: arn:aws:iam::000xxxxxxxxx:mfa/user.name
- 3 for our different environments
- development (acc_no: 111xxxxxxxxx)
role: CrossAccountSignin_Admin: role_arn:aws:iam::111xxxxxxxxx:role/CrossAccountSignin_Admin
- Trusted entities:
acc_no: 000xxxxxxxxx (identity-account)
and Conditions:aws:MultiFactorAuthPresent
- staging (acc_no: 222xxxxxxxxx)
role: CrossAccountSignin_Admin: role_arn:aws:iam::222xxxxxxxxx:role/CrossAccountSignin_Admin
- Trusted entities:
acc_no: 000xxxxxxxxx (identity-account)
and Conditions:aws:MultiFactorAuthPresent
- production (acc_no: 333xxxxxxxxx)
role: CrossAccountSignin_Admin: role_arn:aws:iam::333xxxxxxxxx:role/CrossAccountSignin_Admin
- Trusted entities:
acc_no: 000xxxxxxxxx (identity-account)
and Conditions:aws:MultiFactorAuthPresent
- development (acc_no: 111xxxxxxxxx)
Let’s also assume that the proper roles and policies have already been put in place, allowing us to switch between accounts using the management console. We can create CLI profiles for each of these accounts.
Credentials file ~/.aws/credentials
[piotr-identity-account] aws_access_key_id=333xxxxxxxxxxxxxxxxx aws_secret_access_key=333xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Config file ~/.aws/config
[profile piotr-identity-account] region = eu-west-1 [profile development] region = eu-west-1 source_profile = piotr-identity-account role_arn = arn:aws:iam::111xxxxxxxxx:role/CrossAccountSignin_Admin mfa_serial = arn:aws:iam::000xxxxxxxxx:mfa/piotr [profile staging] region = eu-west-1 source_profile = piotr-identity-account role_arn = arn:aws:iam::222xxxxxxxxx:role/CrossAccountSignin_Admin mfa_serial = arn:aws:iam::000xxxxxxxxx:mfa/piotr [profile production] region = eu-west-1 source_profile = piotr-identity-account role_arn = arn:aws:iam::333xxxxxxxxx:role/CrossAccountSignin_Admin mfa_serial = arn:aws:iam::000xxxxxxxxx:mfa/piotr
- Test
$ aws ec2 describe-instances --profile development Enter MFA code for arn:aws:iam::000xxxxxxxxx:mfa/piotr: ***
By default the temporary API token is valid for 1 hour, once expired you will be prompted for MFA again.
Working with MFA
If you’re using the aws CLI, using MFA with a Credentials Profile is fairly straight forward. You just need to add the mfa_serial to your Config File in ~/.aws/config:
[profile with-mfa] mfa_serial = arn:aws:iam::123456789012:mfa/jon-doe
The mfa_serial parameter should be set to the ARN of your MFA device, which you can get from the IAM User page of the AWS Web Console. It should be of the format:
arn:aws:iam::<ACCOUNT_ID>:mfa/<USERNAME>
Once you’ve added the mfa_serial parameter, the next time you run the aws CLI with that Named Profile, it will prompt you for an MFA token:
$ aws s3 ls --profile with-mfa Enter MFA code:
But what if you’re running something that isn’t aws, such as terraform or packer. Most other tools will not automatically prompt for an MFA token. Instead, you have to do the MFA authentication process outside of that tool. First, you configure your Credentials File with your normal (permanent) AWS Access Keys (e.g. by running aws configure). Next, you run the aws sts get-session-token
command, passing it the ARN of your MFA device and an MFA token from the Google Authenticator App or your key fob:
aws sts get-session-token \ --serial-number arn:aws:iam::123456789012:mfa/jon-doe \ --token-code 123456 \ --duration-seconds 43200
This will return a blob of JSON that contains Temporary Access Keys (note the --duration-seconds
argument in the earlier command, which specifies when these Temporary Access Keys will expire):
{ "Credentials": { "SecretAccessKey": "secret-access-key", "SessionToken": "temporary-session-token", "Expiration": "expiration-date-time", "AccessKeyId": "access-key-id" } }
You will need to take these Temporary Access Keys and copy them into a Named Profile in your Credentials File in ~/.aws/credentials:
[profile-with-mfa] aws_access_key_id = <Access-key-as-in-returned-output> aws_secret_access_key = <Secret-access-key-as-in-returned-output> aws_session_token = <Session-Token-as-in-returned-output>
Note that with Temporary Access Keys, you set not only aws_access_key_id and aws_secret_access_key in your Credentials File, but also aws_session_token
.
Now you can run those other CLI tools with this Named Profile:
export AWS_PROFILE=profile-with-mfa terraform apply
Note that the Temporary Access Keys expire after a certain period of time (typically 12 hours), so you’ll have to do this over again.
Other related AWS cli commands
aws sts get-caller-identity --profile dev #info about you, this will create temp credentials aws sts get-session-token --duration-seconds 28800 #get a session token by default 15 min, here 8h
https://github.com/605data/aws-scripts/tree/master/awscli-mfa https://random.ac/cess/2017/10/28/aws-cli-key-rotation-script-v2/ https://github.com/sweharris/aws-cli-mfa
Assuming role
aws sts assume-role --role-arn "arn:aws:iam::11111111111:role/CrossAccountAgent" --role-session-name "terraform" --profile agent-terraform
Tools making work with sts easier
- aws-vault
- Securely store your AWS credentials in your operating system’s keystore (e.g., Keychain, KWallet), Automatically set those credentials as environment variables when executing a command, Handle all the aws sts commands for you when using IAM Roles or MFA.
- aws-auth
- bash script Gruntwork subscribers only
aws-google-auth
Acquire AWS STS (temporary) credentials via Google Apps SAML Single Sign On
pip install aws-google-auth
Config
$ aws-google-auth Google username: user.name@acme.com Google IDP ID: C00aaabbb Google SP ID: 977777777777 Google Password: ***** Please visit the following URL to view your CAPTCHA: https://accounts.google.com/Captcha?v=2&ctoken=AAAAAAAAA-AAAAAAAAAAAAAAA-BBB(...)-(...)_AAAAAA_AAAAAAA Captcha (case insensitive): fafafafafa Enter SMS token: G-111111 [ 1] arn:aws:iam::111111111111:role/saml-administrator [ 2] arn:aws:iam::222222222222:role/saml-administrator Type the number (1 - 2) of the role to assume: 2 Assuming arn:aws:iam::222222222222:role/saml-administrator Credentials Expiration: 2020-02-12 17:21:51+00:00
Ones setup is completed ~/.aws
credentials files will be updated
cat ~/.aws/config # the following profile will be added [profile sts] region = eu-west-1 google_config.ask_role = False google_config.keyring = False google_config.duration = 43200 google_config.google_idp_id = C00aaabbb google_config.role_arn = arn:aws:iam::222222222222:role/saml-administrator google_config.google_sp_id = 977777777777 google_config.u2f_disabled = False google_config.google_username = user.name@acme.com google_config.bg_response = None cat ~/.aws/credentials [sts] aws_access_key_id = AAAAAAAAAAAAAAAAAAA aws_secret_access_key = bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb aws_security_token = FwoAAAAAAAAAAAAA//////////(...)M0eH aws_session_expiration = 2020-02-12T17:21:51+0000 aws_session_token = FwoAAAAAAAAAAAAA//////////(...)M0eH # The SAML2 response file wll be created cat ~/.aws/saml_cache_C00aaabbb.xml <?xml version="1.0" encoding="UTF-8" standalone="no" ?> <saml2p:Response xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" Destination="https://signin.aws.amazon.com/saml" ... </saml2:Assertion> </saml2p:Response>
okta-aws-cli-assume-role
Install
REFIX=~/.okta bash <(curl -fsSL https://raw.githubusercontent.com/oktadeveloper/okta-aws-cli-assume-role/master/bin/install.sh) -i Warning: Java is not installed. Make sure to install that Installing into ~/.okta Latest release JAR file: https://github.com/oktadeveloper/okta-aws-cli-assume-role/releases/download/v2.0.4/okta-aws-cli-2.0.4.jar Fetching JAR file → ~/.okta/okta-aws-cli-2.0.4.jar Symlinking ~/.okta/okta-aws-cli.jar → okta-aws-cli-2.0.4.jar Creating example ~/.okta/config.properties Add the following to ~/.bash_profile or ~/.profile: #OktaAWSCLI if [[ -f "$HOME/.okta/bash_functions" ]]; then . "$HOME/.okta/bash_functions" fi if [[ -d "$HOME/.okta/bin" && ":$PATH:" != *":$HOME/.okta/bin:"* ]]; then PATH="$HOME/.okta/bin:$PATH" fi
Multi account aws cli execution loop
Helper programme to execute awscli commands across multiple AWS accounts (profiles) and regions.
#Create temp creds for all accounts protected by MFA tokens, run this without timeout command $ # ./awscli-crossaccounts.sh aws sts get-caller-identity #Run a command across accounts in different regions $ # ./awscli-crossaccounts.sh aws ses get-identity-verification-attributes --identities "example.com" #!/bin/bash ##Usage # ./awscli-crossaccounts.sh <full aws cli command> it will be looped through all AWS profiles and regions green_bold="\e[1;32m"; reset="\e[0m" awsprofiles=("default" "dev" "prod") awsregions=("us-east-1" "eu-west-1" "ca-central-1") for profile in "${awsprofiles[@]}"; do for region in "${awsregions[@]}"; do echo -e "${green_bold}$@ --profile $profile --region $region${reset}" #display command to execute timeout 5 "$@" --profile "$profile" --region "$region"; sleep 0.5 #execute, timeout does not allow for interactive MFA session done done
References
- Cli Assume Roles Aws docs
- Roles Terms and Concepts Aws docs!
- Configure AWS cli with Cross-Account Roles & MFA
Examples
Create a reusable delegation set with a unique string '20170409'
aws route53 create-reusable-delegation-set --caller-reference 20170409
List the reusable-delegation-set created in ~/.aws/credentials profile
aws route53 list-reusable-delegation-sets --profile terraform-profile
IAM server certificates
List IAM server certificates, delete a certificate
aws iam list-server-certificates --output text --query 'ServerCertificateMetadataList[*].[Expiration,ServerCertificateName]' | sort aws iam list-server-certificates | grep <ServerCertificateName> aws iam delete-server-certificate --server-certificate-name <ServerCertificateName>
Upload a certificate to IAM
aws iam upload-server-certificate --server-certificate-name cert_name --certificate-body file://cert_name.crt \ --certificate-chain file://cert_name.pem --private-key file://cert_name.key
Check expiry date
certificate_name=<ServerCertificateName> aws iam get-server-certificate --server-certificate-name $certificate_name --output text --query 'ServerCertificate.CertificateBody' | openssl x509 -text | less
List all instances and their status
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[State.Name,InstanceId,Tags[?Key==`Name`].Value]' --output text
Get account number and name (alias)
ACCOUNTID=$(timeout 3 aws sts get-caller-identity --output text | cut -f1 2>/dev/null) ACCO_NAME=$(timeout 3 aws iam list-account-aliases --output text | cut -f2 2>/dev/null)
Tag name in PS1 Bash prompt
Add to your .bashrc, or create a file in /etc/profile.d
INSTANCE_ID=$(wget -qO- http://instance-data/latest/meta-data/instance-id) REGION=$(wget -qO- http://instance-data/latest/meta-data/placement/availability-zone | sed 's/.$//') export PS1="$(aws ec2 describe-tags --region $REGION --filter "Name=resource-id,Values=$INSTANCE_ID" --output=text | cut -f5)$ "
Other variations of export PS1
export PS1="$(aws ec2 describe-tags \ --filters "Name=resource-id,Values=$INSTANCE_ID" 'Name=key,Values=Name' \ --query 'Tags[].Value' --output text)$ " export PS1="$(aws ec2 describe-instances --instance-id $INSTANCE_ID \ --query 'Reservations[].Instances[].Tags[?Key==`Name`].Value' --output text)$ "
Your instance need to have following IAM Profile permissions:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1409309287000", "Effect": "Allow", "Action": [ "ec2:DescribeTags" ], "Resource": [ "*" ] } ] }
AWS Feature request on https://forums.aws.amazon.com - Any plans to make tags available through instance metadata?
References
- AWScli config files
- create-reusable-delegation-set aws . route53 cli
- Configuring White Label Name Servers AWS docs