AWS Basics #4: AWS CLI and SDK Setup

The IAM user’s access keys you created in #2 IAM, plus the billing alerts you turned on in #3. Now it’s time to work with AWS outside the console.

The console is great for learning and one-off work, but for repetition / automation / precise tasks it hits a wall. That’s where two things come in:

• AWS CLI — drive AWS from the terminal with commands like aws s3 cp ...
• SDK — call AWS from inside code (boto3 for Python, aws-sdk for JS, …)

This post covers setup for both, plus the order in which credentials flow behind the scenes (the credential chain).

Installing AWS CLI v2 #

CLI comes in v1 and v2. v2 is the standard — don’t use v1 anymore. What v2 changes:

• Single executable (no Python environment dependency)
• Native IAM Identity Center (SSO) support (#5)
• Faster output / richer autocomplete
• New commands like aws configure import

macOS #

brew install awscli

Or the official pkg installer:

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Linux #

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Windows #

winget or the MSI installer:

winget install -e --id Amazon.AWSCLI

Verifying the install #

aws --version
# aws-cli/2.x.x Python/3.x.x ...

If you don’t see 2.x.x, run which aws and remove any v1 installation.

aws configure — registering credentials #

Once the CLI is installed, register the IAM user’s access keys from #2.

aws configure
# AWS Access Key ID [None]: AKIA...
# AWS Secret Access Key [None]: wJal...
# Default region name [None]: ap-northeast-2
# Default output format [None]: json

Those four values are saved across two files.

~/.aws/credentials   # keys (sensitive)
~/.aws/config        # region / output / role / etc.

Shape of ~/.aws/credentials #

[default]
aws_access_key_id = AKIA...
aws_secret_access_key = wJal...

Shape of ~/.aws/config #

[default]
region = ap-northeast-2
output = json

Profiles — multiple credentials at once #

When you deal with several accounts / environments / roles, a single default isn’t enough. Profiles separate credentials.

Adding profiles #

aws configure --profile dev
aws configure --profile prod

[default]
aws_access_key_id = AKIA...

[dev]
aws_access_key_id = AKIA-DEV-...
aws_secret_access_key = ...

[prod]
aws_access_key_id = AKIA-PROD-...
aws_secret_access_key = ...

Using a profile #

Three ways.

aws s3 ls --profile prod

export AWS_PROFILE=prod
aws s3 ls

function awsenv() {
  export AWS_PROFILE=$1
  echo "AWS_PROFILE=$AWS_PROFILE"
}
# usage: awsenv prod

In production it’s common to use the env var plus a shell prompt that shows the current profile.

Showing the current profile in the prompt #

Showing AWS_PROFILE in the prompt cuts down on incidents.

PROMPT='%n@%m %1~ ${AWS_PROFILE:+[aws:$AWS_PROFILE]} %# '

Coloring the production terminal differently is also common — red for prod, green for dev, etc.

Using a role — the assume-role flow #

Borrowing the role from #2 on the CLI. The standard pattern for multi-account setups.

[profile prod]
role_arn = arn:aws:iam::222222222222:role/AdminRole
source_profile = default
mfa_serial = arn:aws:iam::111111111111:mfa/curtis
region = ap-northeast-2

When you use this profile:

1. Call STS AssumeRole with the default profile’s credentials
2. Prompt for the MFA code (if mfa_serial is set)
3. Receive the temporary credential and use it for the command
4. Cache for 1 hour → reuse

aws s3 ls --profile prod
# Enter MFA code for arn:aws:iam::111111111111:mfa/curtis: 123456
# (cached for 1 hour after that)

In CI environments, IAM Identity Center (SSO) from #5 is smoother.

Environment variables — CI and containers #

Where there’s no ~/.aws/credentials (CI, containers, ephemeral shells), use environment variables.

export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=ap-northeast-2
aws s3 ls

The credential chain — where credentials are looked up #

The CLI / SDK looks up credentials in a fixed order. If you don’t know it, you’ll burn time on “why is it using that key?” debugging.

1. Command-line options  (--profile, --region, etc.)
2. Environment variables (AWS_ACCESS_KEY_ID, etc.)
3. Assist files (CLI Web Identity Token, container credentials)
4. ~/.aws/credentials default or named profile
5. ~/.aws/config profile (role_arn, sso_session, etc.)
6. EC2 / ECS / Lambda instance / container credentials

Why aws s3 ls works inside an EC2 with no configured credentials — step 6, instance metadata, picks it up (the instance profile from #2).

Checking which credential is in use #

aws sts get-caller-identity
# {
#   "UserId": "AIDA...",
#   "Account": "111111111111",
#   "Arn": "arn:aws:iam::111111111111:user/curtis"
# }

This one line is debug-step zero. Which user / which account is doing the work.

Frequently used CLI commands #

Output formats #

aws ec2 describe-instances --output json   # default
aws ec2 describe-instances --output table  # human
aws ec2 describe-instances --output text   # for grep / awk
aws ec2 describe-instances --output yaml   # YAML

–query — picking with JMESPath #

JSON output gets long, and grep alone hits limits. --query extracts only what you need:

aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].[InstanceId,InstanceType]' \
  --output table

JMESPath looks daunting at first, but learning 5–10 patterns will noticeably speed up daily work.

Commands you’ll use often #

aws sts get-caller-identity
aws configure list
aws configure list-profiles

aws s3 ls                              # all buckets
aws s3 ls s3://my-bucket/              # inside a bucket
aws s3 cp file.txt s3://my-bucket/     # upload
aws s3 sync ./local s3://my-bucket/    # folder sync
aws s3 rm s3://my-bucket/file.txt      # delete
aws s3 presign s3://my-bucket/file.pdf --expires-in 3600

aws ec2 describe-instances
aws ec2 start-instances --instance-ids i-...
aws ec2 stop-instances --instance-ids i-...
aws ec2 describe-regions --query 'Regions[].RegionName'

aws iam list-users
aws iam get-user --user-name curtis
aws iam list-attached-user-policies --user-name curtis
aws iam create-access-key --user-name curtis

SDK — AWS inside your code #

If the CLI is the terminal, the SDK is your code. They share the same credential chain.

Python — boto3 #

pip install boto3

import boto3

s3 = boto3.client("s3")
res = s3.list_buckets()
for b in res["Buckets"]:
    print(b["Name"])

Credentials come automatically from the credential chain — ~/.aws/credentials, env vars, EC2 instance profile, and so on.

Explicit profile / region #

session = boto3.Session(profile_name="prod", region_name="ap-northeast-2")
s3 = session.client("s3")

Resource vs client #

boto3 has two interfaces.

s3 = boto3.client("s3")
res = s3.list_objects_v2(Bucket="my-bucket")
for obj in res.get("Contents", []):
    print(obj["Key"])

s3 = boto3.resource("s3")
for obj in s3.Bucket("my-bucket").objects.all():
    print(obj.key)

Modern boto3 recommends client. Resource only supports some services, and new services aren’t being added.

Pagination — a frequent pitfall #

list_objects_v2 returns at most 1000 entries per call. For more, use a paginator.

s3 = boto3.client("s3")
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-bucket"):
    for obj in page.get("Contents", []):
        print(obj["Key"])

JavaScript / TypeScript — aws-sdk v3 #

v2 is deprecated. Go with v3.

npm install @aws-sdk/client-s3

import { S3Client, ListBucketsCommand } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-2" });
const res = await client.send(new ListBucketsCommand({}));
for (const b of res.Buckets ?? []) {
  console.log(b.Name);
}

What v3 changes:

• Modular — import only the services you need (smaller bundles)
• Tree-shakable
• Every call follows client.send(new XxxCommand(...))

Go — aws-sdk-go-v2 #

go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/s3

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    ctx := context.Background()
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("ap-northeast-2"))
    if err != nil {
        panic(err)
    }
    client := s3.NewFromConfig(cfg)
    res, err := client.ListBuckets(ctx, &s3.ListBucketsInput{})
    if err != nil {
        panic(err)
    }
    for _, b := range res.Buckets {
        fmt.Println(*b.Name)
    }
}

The conventions of the Go track — context first plus explicit errors.

Turning on autocomplete #

Enabling tab completion makes the CLI a lot smoother.

autoload bashcompinit && bashcompinit
autoload -Uz compinit && compinit
complete -C "$(which aws_completer)" aws

complete -C "$(which aws_completer)" aws

Drop into .zshrc / .bashrc and start a new shell. Now aws s3 <TAB> shows commands / options.

Common pitfalls #

1) Committing ~/.aws/credentials to git #

The most dangerous incident. Always add .aws/ to .gitignore or your global gitignore. Better still: keep keys out of files entirely — env vars / SSO are increasingly the standard.

2) Running prod commands under the wrong profile #

You’re working with AWS_PROFILE=dev and run aws s3 rb s3://...-prod — the env var says dev, yet the prod bucket is targeted. Turns out a separate --profile prod was on the command. Always verify first with aws sts get-caller-identity before running anything destructive.

3) Debugging the credential chain #

“I set the keys in env vars but it’s still using an old key” — chain step 1 (the --profile flag) wins. Or your shell didn’t pick up the export. aws sts get-caller-identity is the first diagnostic.

4) v1 leftovers #

aws --version shows 1.x (often left over from pip install awscli on macOS / Linux). With both v1 and v2 in PATH you can end up running v1 unintentionally — use which aws to confirm which binary is active and remove the old one.

5) Missing pagination #

Almost every list_* API paginates. Process only the first page and you’ll miss everything past entry 1000 in production data. Use paginator or a NextToken loop.

6) Hard-coding credentials in the SDK #

boto3.client("s3", aws_access_key_id="AKIA...", aws_secret_access_key="...")

Credentials come from the environment / profile / instance profile. Hard-code them and they end up in git → leak → incident.

Wrap-up #

What we covered:

• Install AWS CLI v2 — Homebrew / curl / winget. v1 is no longer in use
• aws configure registers credentials → split across ~/.aws/credentials + ~/.aws/config
• Profiles separate accounts / environments. --profile, AWS_PROFILE, shell functions
• Role profiles for the AssumeRole flow. role_arn + source_profile + mfa_serial
• Env vars — CI / containers / ephemeral shells. Beyond AWS_ACCESS_KEY_ID, also AWS_SESSION_TOKEN, AWS_REGION
• Credential chain — command line → env → file → instance profile. aws sts get-caller-identity is debug-step zero
• CLI options — --output table/json/text, --query (JMESPath)
• SDK — boto3 / aws-sdk v3 / aws-sdk-go-v2. Shared credential chain, client / resource, paginator
• Pitfalls — committing credentials, mixed-up profiles, v1 leftovers, missing pagination, keys in code

Next — CloudShell and SSO #

Local setup is done. But sometimes you need the same CLI on someone else’s laptop / a temporary work environment. And once multi-account starts, the login itself has to get more sophisticated.

#5 CloudShell and IAM Identity Center (SSO) covers CloudShell (the in-browser terminal) plus IAM Identity Center (SSO) — the standard login for multi-account.