Getting started with AWS

This document provides instructions on installing the AWS CLI tool and setting up credentials on your local machine.

Installing AWS CLI

This section provides instructions to install the AWS CLI tool.

For more details or troubleshooting, please refer to the the official instructions for installing the AWS CLI.

We attempt to summarize the above with a series of commands that should “just work” and install the aws tool on your machine. This does assume a Linux Ubuntu machine with an x86_64 processor. For other systems refer to the official docs.

It is also important to have the curl, zip, and gpg packages installed.

# Ensure you have curl on your system
dpkg -l curl > /dev/null || sudo apt install curl -y

# Ensure unzip is installed
dpkg -l zip > /dev/null || sudo apt install zip -y

# Ensure gpg is installed
dpkg -l gnupg > /dev/null || sudo apt install gnupg -y

We recommend running the following in a temporary directory.

mkdir -p $HOME/tmp
cd $HOME/tmp

The first step is to download the aws cli tool.

# Download the CLI tool for linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscli-exe-linux-x86_64.zip"

The next step is to verify the integrity of the downloaded tool. This step can be skipped, but it is usually a good idea to do this.

# Import the amazon GPG public key
echo "
    -----BEGIN PGP PUBLIC KEY BLOCK-----

    mQINBF2Cr7UBEADJZHcgusOJl7ENSyumXh85z0TRV0xJorM2B/JL0kHOyigQluUG
    ZMLhENaG0bYatdrKP+3H91lvK050pXwnO/R7fB/FSTouki4ciIx5OuLlnJZIxSzx
    PqGl0mkxImLNbGWoi6Lto0LYxqHN2iQtzlwTVmq9733zd3XfcXrZ3+LblHAgEt5G
    TfNxEKJ8soPLyWmwDH6HWCnjZ/aIQRBTIQ05uVeEoYxSh6wOai7ss/KveoSNBbYz
    gbdzoqI2Y8cgH2nbfgp3DSasaLZEdCSsIsK1u05CinE7k2qZ7KgKAUIcT/cR/grk
    C6VwsnDU0OUCideXcQ8WeHutqvgZH1JgKDbznoIzeQHJD238GEu+eKhRHcz8/jeG
    94zkcgJOz3KbZGYMiTh277Fvj9zzvZsbMBCedV1BTg3TqgvdX4bdkhf5cH+7NtWO
    lrFj6UwAsGukBTAOxC0l/dnSmZhJ7Z1KmEWilro/gOrjtOxqRQutlIqG22TaqoPG
    fYVN+en3Zwbt97kcgZDwqbuykNt64oZWc4XKCa3mprEGC3IbJTBFqglXmZ7l9ywG
    EEUJYOlb2XrSuPWml39beWdKM8kzr1OjnlOm6+lpTRCBfo0wa9F8YZRhHPAkwKkX
    XDeOGpWRj4ohOx0d2GWkyV5xyN14p2tQOCdOODmz80yUTgRpPVQUtOEhXQARAQAB
    tCFBV1MgQ0xJIFRlYW0gPGF3cy1jbGlAYW1hem9uLmNvbT6JAlQEEwEIAD4WIQT7
    Xbd/1cEYuAURraimMQrMRnJHXAUCXYKvtQIbAwUJB4TOAAULCQgHAgYVCgkICwIE
    FgIDAQIeAQIXgAAKCRCmMQrMRnJHXJIXEAChLUIkg80uPUkGjE3jejvQSA1aWuAM
    yzy6fdpdlRUz6M6nmsUhOExjVIvibEJpzK5mhuSZ4lb0vJ2ZUPgCv4zs2nBd7BGJ
    MxKiWgBReGvTdqZ0SzyYH4PYCJSE732x/Fw9hfnh1dMTXNcrQXzwOmmFNNegG0Ox
    au+VnpcR5Kz3smiTrIwZbRudo1ijhCYPQ7t5CMp9kjC6bObvy1hSIg2xNbMAN/Do
    ikebAl36uA6Y/Uczjj3GxZW4ZWeFirMidKbtqvUz2y0UFszobjiBSqZZHCreC34B
    hw9bFNpuWC/0SrXgohdsc6vK50pDGdV5kM2qo9tMQ/izsAwTh/d/GzZv8H4lV9eO
    tEis+EpR497PaxKKh9tJf0N6Q1YLRHof5xePZtOIlS3gfvsH5hXA3HJ9yIxb8T0H
    QYmVr3aIUes20i6meI3fuV36VFupwfrTKaL7VXnsrK2fq5cRvyJLNzXucg0WAjPF
    RrAGLzY7nP1xeg1a0aeP+pdsqjqlPJom8OCWc1+6DWbg0jsC74WoesAqgBItODMB
    rsal1y/q+bPzpsnWjzHV8+1/EtZmSc8ZUGSJOPkfC7hObnfkl18h+1QtKTjZme4d
    H17gsBJr+opwJw/Zio2LMjQBOqlm3K1A4zFTh7wBC7He6KPQea1p2XAMgtvATtNe
    YLZATHZKTJyiqA==
    =vYOk
    -----END PGP PUBLIC KEY BLOCK-----
" | sed -e 's|^ *||' > aws.pub
cat aws.pub
gpg --import aws.pub

# Download the signature and verify the CLI tool is signed by amazon
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip.sig" -o "awscli-exe-linux-x86_64.zip.sig"

gpg --verify awscli-exe-linux-x86_64.zip.sig awscli-exe-linux-x86_64.zip

Now that we have verified the integrity, install the aws CLI tool to your local PATH.

# Unzip the downloaded installer
unzip -o awscli-exe-linux-x86_64.zip

# If you want to install somewhere else, change the PREFIX variable
PREFIX="$HOME/.local"
mkdir -p $PREFIX/bin
./aws/install --install-dir $PREFIX/aws-cli --bin-dir $PREFIX/bin --update

Note the value of PREFIX in the above step. The directory $PREFIX/bin should be in your PATH. If you do not have that location in your path we recommend adding it like this:

# Add to the path in the current shell
export PATH=$HOME/.local/bin/:$PATH

# Add the line to your bashrc so all new shells will have the local bin in
# your path
echo 'export PATH=$HOME/.local/bin/:$PATH' >> $HOME/.bashrc

Test that your new AWS CLI is working by running:

aws --version

Now that you have the AWS CLI, the next step is to ensure you have the correct credentials.

AWS Credentials

This document is designed for internal collaberators and will provide instructions on setting up credentials for an IARPA profile, which will give you access to the SMART S3 buckets.

To use the AWS CLI (and by extension a DVC AWS remote), you must have credentials and a config. The default location to store credentials is: $HOME/.aws/credentials The default location to store a config is: $HOME/.aws/config.

Obtaining Credentials

To obtain credentials, the current point of contact is yonatan.gefen@kitware.com (as of 2022-10-06). Please send Yoni an email and CC matt.leotta@kitware.com and jon.crall@kitware.com to request credentials.

We will then start the process of securely sending you your credentials. If you have a public GPG key, please send that with your request. We will encrypt your credentials with your GPG public key, send it to you, and then only you can decrypt it with your GPG private key.

If you don’t have a GPG we will use manual Diffie Hellman handshake. Navigate to https://cryptotools.net/dhe and generate a private and public key. Send the public key in your email (don’t leave this page, until the process is done). We will then do the same process on our end, and we will send you our public key. The next step is we will both paste each other’s public keys into the webpage which will establish a shared secret key. Copy down this shared key, you will need it later.

On our end, we will take your credentials and encrypt them with this shared secret. We will send you the encrypted data. Then navigate to https://cryptotools.net/aes, click “decrypt”, paste in both the shared secret key and then the encrypted message. The plaintext credentials will be generated in the top box. These are your credentials that we will use in the subsequent steps.

To summarize, here is an example. Alice wants to send Bob the secret message: “hello world”.

  • Alice navigates to https://cryptotools.net/dhe, generates a public key: bSoNKmm2qF2HLo2tG39gVN4c5xuMnBqX6ES4C0nLdOI=, and sends it to Bob.

  • Bob navigates to https://cryptotools.net/dhe, generates a public key: UYXjuE9QpXASQM8QQmjImECyvIg4MsOwkS3YrTXXLB0=, and sends it to Alice.

  • Alice enters Bob’s secret key into her “Public key” on the right.

  • Bob enters Alices’s secret key into his “Public key” on the right.

  • Both Alice and Bob now see a shared secret: arnE9PLCOHrvKRLAXsrx+Nc4pyCBZtjCoESjo16Fvi8= appear, which they can now use for encryption and decryption.

  • Alice navigates to https://cryptotools.net/aes, enters the plain text “hello world” and uses the shared secret arnE9PLCOHrvKRLAXsrx+Nc4pyCBZtjCoESjo16Fvi8= as the encryption key. This generates the encrypted cyphertext U2FsdGVkX19sofdkwHQvnur20N8KwDULOxqVPkboYxI=, which Alice can send to Bob.

  • Bob receives the cyphertext from Alice, navigates to https://cryptotools.net/aes, and hits the “Decrypt” button. He enters the cyphertext U2FsdGVkX19sofdkwHQvnur20N8KwDULOxqVPkboYxI= into the bottom pane, and also enters the shared secret arnE9PLCOHrvKRLAXsrx+Nc4pyCBZtjCoESjo16Fvi8= into the key feild. The decryption happens automatically and the secret message appears in the top plaintext box.

Using Credentials

In the credentials file ($HOME/.aws/credentials) append the following text to create credentials associated with the “iarpa” AWS_PROFILE.

[iarpa]
aws_access_key_id = <YOUR_ACCESS_KEY>
aws_secret_access_key = <YOUR_SECRET_KEY>

For the config file ($HOME/.aws/config), it is important to specify the region for the iarpa profile. Set output to either text or json.

[profile iarpa]
region=us-west-2
output=json

That completes the install. Verify that it worked by attempting to access bucket containing the DVC cache:

aws --profile iarpa s3 ls s3://kitware-smart-watch-data/dvc/

Note the --profile iarpa tells aws to authenticate using the “iarpa” profile in our config/credentials. We could also set an environment variable export AWS_PROFILE=iarpa.

The contents of that folder will be a long list of 2 letter folders and temp files. This is the hashed file structure that the dvc cache uses. include a “dvc” directory. Seeing this means that you authenticated sucessfully. Note that when working with DVC you will not need to use the cache directly, we are simply checking that you have access to it.

AWS Security

It is important to periodically rotate your AWS credentials.

See detailed internal instructions for rotating keys: here.

# Install the AWS key rotation script

[[ -d $HOME/code/aws-rotate-iam-keys ]] || git clone https://github.com/rhyeal/aws-rotate-iam-keys.git $HOME/code/aws-rotate-iam-keys
cp $HOME/code/aws-rotate-iam-keys/src/bin/aws-rotate-iam-keys $HOME/.local/bin

cat $HOME/.aws/config
cat $HOME/.aws/credentials

# Execute key rotation on your local machine on the IARPA profile
export AWS_PROFILE=iarpa
aws-rotate-iam-keys --profile $AWS_PROFILE

# Synchronize those keys to all other machine that need them.
# Doing this will depend on how the user synchronizes keys.

Next Steps

  • getting started with kubectl <getting_started_kubectl.rst>

  • getting started with dvc <getting_started_dvc.rst>