Of course we cannot always share details about our work with customers, but nevertheless it is nice to show our technical achievements and share some of our implemented solutions.
Some of our EC2 instances, used as Docker hosts, started to use up the instances capacities. Time to upgrade to a new instance.
It's 2019 and wow, there are so many new instance types. It feels like being in a candy shop where all you see is candy and you want to try them all. But you forget about the price tag.
Well to narrow down the list, I compared some of the newer instances with the type we currently run (m4.xlarge):
| Instance name |
vCPU |
ECU |
Memory |
Storage |
Pricing (Linux) |
| m4.xlarge | 4 | 13 | 16 GB |
EBS Only |
$0.222 / h |
| m4.2xlarge |
8 |
26 |
32 GB |
EBS Only | $0.444 / h |
| m5.2xlarge |
8 |
31 |
32 GB |
EBS Only |
$0.428 / h |
| m5a.2xlarge |
8 |
N/A |
32 GB |
EBS Only |
$0.384 / h |
| t2.2xlarge |
8 |
Variable |
32 GB |
EBS Only |
$0.4032 / h |
| t3.2xlarge |
8 |
Variable |
32 GB |
EBS Only |
$0.3648 / h |
By choosing the "logical" bigger instance (m4.2xlarge) we would pay exactly the double per hour. But the t3.2xlarge instance type caught my attention as it offers the same capacity specs but is cheaper than the m4 equivalent. The instance type description explain the difference between M and T instances this way:
M4/M5 instances provide a balance of compute, memory, and network resources, and it is a good choice for many applications.
T3 instances accumulate CPU credits when a workload is operating below
baseline threshold. Each earned CPU credit provides the T3 instance the
opportunity to burst with the performance of a full CPU core for one
minute when needed.
This does sound interesting and is actually what we need. We see CPU spikes whenever a deployment happens and new containers are rolled out/upgraded. However during "normal runtime" the CPU usage is pretty steady.
After shutdown of the instance, I changed the instance type and wanted to start it. But got this error message popping up instead:
Enhanced networking with the Elastic Network Adapter (ENA) is required for the 't3.2xlarge' instance type. Ensure that your instance 'i-073XXXXXXXXXXXXXX' is enabled for ENA.
I quickly read through the "Enabling enhanced networking with the Elastic network adapter" guide and verified on another still running instance if the Kernel module exists in that Ubuntu (16.04) image:
root@aws-ec2-instance:~# modinfo ena
filename: /lib/modules/4.4.0-142-generic/kernel/drivers/net/ethernet/amazon/ena/ena.ko
version: 2.0.2K
license: GPL
description: Elastic Network Adapter (ENA)
author: Amazon.com, Inc. or its affiliates
srcversion: 9C8CC5ABAE1D49A2C13BCDC
alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
depends:
retpoline: Y
intree: Y
vermagic: 4.4.0-142-generic SMP mod_unload modversions retpoline
parm: debug:Debug level (0=none,...,16=all) (int)
If this Kernel module would not be on the system, the instance would (most likely) not start up correctly.
But a click on "Yes, Start" didn't help. This message didn't go aways. What did I miss?
Obviously my "quick read" led me into thinking that only the Linux Kernel must be ready for the new interface type. However this is not enough. The AWS EC2 instance itself needs to be told it is ENA-ready as well. Technically speaking: An additional attribute "EnaSupport" needs to be added to the instance. But this can only be done using the AWS CLI, not through the UI.
Luckily this wasn't something new but for sake of completeness here's a quick guide how to install and configure the AWS cli.
First your own user needs to have an API key, which can be managed through the "IAM" user management in the AWS console.
Install the AWS cli:
admck@WM2856l ~ $ sudo apt-get install awscli
Then set up your local AWS configuration using the API key and its secret (you got this from IAM before):
admck@WM2856l ~ $ aws configure
AWS Access Key ID [None]: YOUR-OWN-KEY-ID
AWS Secret Access Key [None]: YOUR-KEYS-SECRET
Default region name [None]: eu-central-1
Default output format [None]:
Now, using the cli, certain commands can be run on the instances. For example to retrieve some values:
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX --query "Reservations[].Instances[].EbsOptimized"
[
true
]
So far so good. Let's check for the "EnaSupport":
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX --query "Reservations[].Instances[].EnaSupport"
admck@WM2856l ~ $ echo $?
0
Oh... this field doesn't seem to exist, so this instance is not ready yet.
To enable ENA support on this instance, according to the ENA documentation:
admck@WM2856l ~ $ aws ec2 modify-instance-attribute --instance-id i-073XXXXXXXXXXXXXX --ena-support
Unknown options: --ena-support
Oh come on, what now?
The awscli package installed the AWS cli in version 1.2.9:
admck@WM2856l ~ $ dpkg -l|grep awscli
ii awscli 1.2.9-2 all Universal Command Line Environment for AWS
What's the current version? According to the current release notes it is 1.10.38. Wow, the local version is way behind!
To install the current version, the awscli package should be removed and the current version installed using pip:
admck@WM2856l ~ $ sudo apt-get remove awscli
admck@WM2856l ~ $ sudo pip install awscli
Let's try an AWS cli command again using the now up-to-date version:
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX --query "Reservations[].Instances[].EnaSupport"
/usr/local/lib/python2.7/dist-packages/urllib3/util/ssl_.py:365: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
SNIMissingWarning
[]
Oh, come on! This error seems to be caused by a bug in older python versions. This machine still uses a pretty old python version:
admck@WM2856l ~ $ python -V
Python 2.7.6
It is suggested to upgrade to python >= 2.7.9. However, as this machine also has the python3 packages installed, I'm trying to use the (parallel) installed python3:
admck@WM2856l ~ $ python3 -V
Python 3.4.3
The aws command itself is a Python script, so the handler can be changed:
admck@WM2856l ~ $ head /usr/local/bin/aws
#!/usr/bin/python
# Copyright 2012 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
# http://aws.amazon.com/apache2.0/
# or in the "license" file accompanying this file. This file is
By adjusting the command handler from #!/usr/bin/python to #!/usr/bin/python3, python3 should be used to run the script.
admck@WM2856l ~ $ head /usr/local/bin/aws
#!/usr/bin/python3
# Copyright 2012 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
# http://aws.amazon.com/apache2.0/
# or in the "license" file accompanying this file. This file is
OK, it should work now, right?
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX
Traceback (most recent call last):
File "/usr/local/bin/aws", line 19, in
import awscli.clidriver
ImportError: No module named 'awscli'
But that was of course my own fault because awscli needs to be installed as pip for python3 (pip3), too:
admck@WM2856l ~ $ sudo pip3 install awscli
Testing again with the previous check for EbsOptimized:
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX --query "Reservations[].Instances[].EbsOptimized"
[
true
]
Finally, now the "aws" command runs with python3. Let's try to set the EnaSupport again:
admck@WM2856l ~ $ aws ec2 modify-instance-attribute --instance-id i-073XXXXXXXXXXXXXX --ena-support
admck@WM2856l ~ $ echo $?
0
No error this time. Let's verify:
admck@WM2856l ~ $ aws ec2 describe-instances --instance-id i-073XXXXXXXXXXXXXX --query "Reservations[].Instances[].EnaSupport"
[
true
]
Yes! Now it finally worked and EnaSupport is enabled on this instance!
Now that the aws cli is anyway in use, the instance can also be started through the cli:
admck@WM2856l ~ $ aws ec2 start-instances --instance-id i-073XXXXXXXXXXXXXX
{
"StartingInstances": [
{
"CurrentState": {
"Code": 0,
"Name": "pending"
},
"PreviousState": {
"Code": 80,
"Name": "stopped"
},
"InstanceId": "i-073XXXXXXXXXXXXXX"
}
]
}
The instance started successful and now runs with the new instance type t3.2xlarge. Which comes with NVMe drives by the way. Sweet.