AWS CLI and SDK Setup

The access key of the user you made in Chapter 2 IAM, and the billing alerts you turned on in Chapter 3 cost management, are ready. Now it’s time to work with AWS outside the console.

The console is good for learning or one-off work, but it hits a limit for repetitive / automated / precise work. That’s when two things show up. One is the AWS CLI, a tool to drive AWS from the terminal with commands like aws s3 cp .... The other is an SDK, a library to call AWS from inside code (Python’s boto3, JS’s aws-sdk, etc.).

This chapter sorts out the setup of those two, and the order credentials flow behind them (the credential chain). The credential chain you grasp here carries straight through to the SSO login of Chapter 5 CloudShell and SSO.

Installing AWS CLI v2 #

The CLI has a v1 and a v2. v2 is the standard. v1 is no longer used. The differences of v2:

• A single executable (no Python-environment dependency)
• Native support for IAM Identity Center (SSO) (Chapter 5 CloudShell and SSO)
• Faster output and richer autocompletion
• New commands like aws configure import

macOS #

brew install awscli

Or install with the official pkg.

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Linux #

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Windows #

Use winget or the MSI installer.

winget install -e --id Amazon.AWSCLI

Verifying the install #

aws --version
# aws-cli/2.x.x Python/3.x.x ...

If you don’t see 2.x.x, check for v1 leftovers with which aws and remove them.

aws configure — registering credentials #

Once the CLI is installed, register the user access key from Chapter 2 IAM.

aws configure
# AWS Access Key ID [None]: AKIA...
# AWS Secret Access Key [None]: wJal...
# Default region name [None]: ap-northeast-2
# Default output format [None]: json

These four values are stored split across two files.

~/.aws/credentials   # keys (sensitive)
~/.aws/config        # region / output / role / etc.

The shape of ~/.aws/credentials #

[default]
aws_access_key_id = AKIA...
aws_secret_access_key = wJal...

The shape of ~/.aws/config #

[default]
region = ap-northeast-2
output = json

Profiles — several credentials at once #

When you handle multiple accounts / environments / roles, one default isn’t enough. Separate several credentials with profiles.

Adding a profile #

aws configure --profile dev
aws configure --profile prod

[default]
aws_access_key_id = AKIA...

[dev]
aws_access_key_id = AKIA-DEV-...
aws_secret_access_key = ...

[prod]
aws_access_key_id = AKIA-PROD-...
aws_secret_access_key = ...

Using a profile #

There are three ways.

aws s3 ls --profile prod

export AWS_PROFILE=prod
aws s3 ls

function awsenv() {
  export AWS_PROFILE=$1
  echo "AWS_PROFILE=$AWS_PROFILE"
}
# Usage: awsenv prod

In operations, a common pattern is, along with an environment variable, showing the current profile in the shell prompt.

Showing the current profile in the shell prompt #

Showing AWS_PROFILE in the prompt reduces incidents.

PROMPT='%n@%m %1~ ${AWS_PROFILE:+[aws:$AWS_PROFILE]} %# '

Making the production terminal a different color is also a common pattern. Set red for the prod profile, green for dev, and so on.

Using a Role — the assume-role flow #

The flow for borrowing the role covered in Chapter 2 IAM from the CLI. The standard pattern for a multi-account setup.

[profile prod]
role_arn = arn:aws:iam::222222222222:role/AdminRole
source_profile = default
mfa_serial = arn:aws:iam::111111111111:mfa/curtis
region = ap-northeast-2

When you use this profile, it works in the following order.

1. It calls STS AssumeRole with the credentials of the default profile.
2. It asks for the MFA code (if mfa_serial is present).
3. It receives temporary credentials and uses them in the command.
4. It caches them for an hour for reuse.

aws s3 ls --profile prod
# Enter MFA code for arn:aws:iam::111111111111:mfa/curtis: 123456
# (cached for an hour afterward)

In a CI environment, the IAM Identity Center (SSO) of Chapter 5 CloudShell and SSO is smoother.

Environment variables — credentials for CI and containers #

In environments without ~/.aws/credentials (CI, containers, an ephemeral shell), use environment variables.

export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=ap-northeast-2
aws s3 ls

The credential chain — where credentials are looked for #

The order the CLI and SDK look for credentials is fixed. Not knowing it wastes your time on “why is it running with this key?” debugging.

1. Command-line options  (--profile, --region, etc.)
2. Environment variables (AWS_ACCESS_KEY_ID, etc.)
3. Assistant files       (CLI Web Identity Token, Container Credentials)
4. The default or specified profile in ~/.aws/credentials
5. Profiles in ~/.aws/config (role_arn, sso_session, etc.)
6. Instance / container credentials of EC2 / ECS / Lambda

The reason aws s3 ls works without credentials inside EC2 is that step 6’s instance metadata catches it (Chapter 2 IAM’s instance profile).

Checking which credential is used #

aws sts get-caller-identity
# {
#   "UserId": "AIDA...",
#   "Account": "111111111111",
#   "Arn": "arn:aws:iam::111111111111:user/curtis"
# }

This is the first step of debugging. It tells you which user and which account you’re running as.

Commonly used CLI commands #

Output formats #

aws ec2 describe-instances --output json   # default
aws ec2 describe-instances --output table  # for humans
aws ec2 describe-instances --output text   # for grep / awk
aws ec2 describe-instances --output yaml   # YAML

–query — picking out with JMESPath #

JSON output is long, so grep hits a limit. Pick out just the part you need with --query.

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].[InstanceId,InstanceType]' \
  --output table

JMESPath syntax is hard at first, but memorize just 5 ~ 10 commonly used patterns and day-to-day work gets faster.

Commonly used commands #

aws sts get-caller-identity
aws configure list
aws configure list-profiles

aws s3 ls                              # all buckets
aws s3 ls s3://my-bucket/              # inside a bucket
aws s3 cp file.txt s3://my-bucket/     # upload
aws s3 sync ./local s3://my-bucket/    # sync a folder
aws s3 rm s3://my-bucket/file.txt      # delete
aws s3 presign s3://my-bucket/file.pdf --expires-in 3600

aws ec2 describe-instances
aws ec2 start-instances --instance-ids i-...
aws ec2 stop-instances --instance-ids i-...
aws ec2 describe-regions --query 'Regions[].RegionName'

aws iam list-users
aws iam get-user --user-name curtis
aws iam list-attached-user-policies --user-name curtis
aws iam create-access-key --user-name curtis

SDK — AWS inside code #

If the CLI is the terminal, the SDK is code. It shares the same credential chain.

Python — boto3 #

pip install boto3

import boto3

s3 = boto3.client("s3")
res = s3.list_buckets()
for b in res["Buckets"]:
    print(b["Name"])

Credentials are found automatically from the credential chain — ~/.aws/credentials, environment variables, the EC2 instance profile, etc.

Explicit profile / Region #

session = boto3.Session(profile_name="prod", region_name="ap-northeast-2")
s3 = session.client("s3")

Resource vs client #

boto3 has two interfaces.

s3 = boto3.client("s3")
res = s3.list_objects_v2(Bucket="my-bucket")
for obj in res.get("Contents", []):
    print(obj["Key"])

s3 = boto3.resource("s3")
for obj in s3.Bucket("my-bucket").objects.all():
    print(obj.key)

Modern boto3 recommends client. resource supports only some services, and new services are no longer added.

Pagination — a common pitfall #

list_objects_v2 returns at most 1000 at a time. For more than that, receive them with a paginator.

s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-bucket"):
    for obj in page.get("Contents", []):
        print(obj["Key"])

JavaScript / TypeScript — aws-sdk v3 #

v2 is deprecated. Go with v3.

npm install @aws-sdk/client-s3

import { S3Client, ListBucketsCommand } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-2" });
const res = await client.send(new ListBucketsCommand({}));
for (const b of res.Buckets ?? []) {
  console.log(b.Name);
}

The traits of v3:

• Modular — import only the services you need (smaller bundle size)
• Tree-shaking friendly
• Every call follows the client.send(new XxxCommand(...)) pattern

Go — aws-sdk-go-v2 #

go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/s3

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    ctx := context.Background()
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("ap-northeast-2"))
    if err != nil {
        panic(err)
    }
    client := s3.NewFromConfig(cfg)
    res, err := client.ListBuckets(ctx, &s3.ListBucketsInput{})
    if err != nil {
        panic(err)
    }
    for _, b := range res.Buckets {
        fmt.Println(*b.Name)
    }
}

It follows context-first and explicit errors, per the conventions of the Go track.

Turning on autocompletion #

Turning on tab autocompletion makes the CLI much smoother.

autoload bashcompinit && bashcompinit
autoload -Uz compinit && compinit
complete -C "$(which aws_completer)" aws

complete -C "$(which aws_completer)" aws

Put it in .zshrc or .bashrc and open a new shell. Now aws s3 <TAB> shows commands and options.

Common pitfalls #

• Committing ~/.aws/credentials to git — the most dangerous incident. Always put .aws/ in .gitignore or in a global gitignore. And keeping access keys in environment variables / SSO rather than in a file is increasingly the standard.
• Operational commands on the wrong profile — you’re working with AWS_PROFILE=dev but a command has --profile prod written separately, so a prod bucket shows up. The answer is to make the source explicit and check first with aws sts get-caller-identity.
• Credential-chain debugging — if “I put a key in an environment variable but it’s running with the old key,” it’s likely that credential-chain step 1 (command-line --profile) takes priority, or the shell didn’t pick up the export. Diagnose first with aws sts get-caller-identity.
• v1 leftovers — if aws --version is 1.x (especially traces of pip install awscli on macOS / Linux), it’s a v1 leftover. If v1 and v2 are both on PATH, an unintended v1 runs, so check with which aws and remove it.
• Missing pagination — almost all list_* APIs have pagination. Process only the first page and you miss data past the 1000th. Use a paginator or a NextToken loop.
• Putting the SDK’s credentials in code — putting keys in code like below leads to exposure via git and an incident.

boto3.client("s3", aws_access_key_id="AKIA...", aws_secret_access_key="...")

Credentials are received from the environment / profile / instance profile.

Exercises #

1. Without looking, write the six steps of §“The credential chain” in order. Then explain, by connecting to Chapter 2 IAM’s instance profile, which step is the reason aws s3 ls works inside EC2 without a key.
2. Explain in one paragraph which flow of Chapter 2 IAM’s Trust Policy and Permission Policy the role_arn + source_profile + mfa_serial profile in §“Using a Role — the assume-role flow” reproduces.
3. Referring to the table in §"–query," write a --query expression yourself that pulls only the ID and type of running instances from the output of aws ec2 describe-instances.

In short: AWS CLI v2 and the SDK share the same credential chain, looking for credentials in the order command line → environment variables → file → instance profile. aws configure stores credentials across ~/.aws/credentials and ~/.aws/config, and separates multiple accounts with profiles. Not putting keys in code or git, and checking identity first with aws sts get-caller-identity, is the basis of debugging and incident prevention.

Next chapter #

The local setup is done. But there are times you need the same CLI on someone else’s laptop or in a company’s temporary environment, and once multi-account begins, login itself has to get more refined. In the next Chapter 5 CloudShell and SSO, we sort out the in-console browser terminal CloudShell, and the IAM Identity Center (SSO) setup that has become the standard login for multi-account.