Create an EC2 Instance

AMI Origin

The ami for our EC2 instances was originally created a long time ago using Packer (in the os-images repo).

This ami provides a basic setup that has worked for us for the last few years, including user setups and basic Debian configuration.

BUT! The original creation script doesn’t seem to work anymore and servers are now created by 'copying' one of the existing servers, and some subsequent manual faffing.

Rather than fix this, the intention is to stop maintaining so many servers, and move everything over to AWS Fargate instead. Whatever scripts we use to manage that process - which is unlikely to be Ansible - should then also be used to generate the few EC2 servers we will need, such as for persistent database.

Create new Server

In the EC2 console select an existing server that matches the size you need.

Then select Launch more like this. (This is currently under Actions > Images and Templates.)

You should have an smc_general_host security group, which will do for starters in most scenarios. This assumes that only SSH and HTTP(s) access is required, and that nginx will route internally to any Docker containers.

Verify your storage options are correct. All our servers have a separate volume for the /var directory at least. See the section below if you need to resize this volume.

Make sure you update the tags so you can tell this server apart from the one you just copied.

DNS

Point the <hostname>.dgcsdev.com Route53 record to your new server’s public IP.

Then add *.<hostname>.dgcsdev.com as an alias to this DNS record.

SSH Keys

The keys baked into this image are old and have been compromised. You MUST change them immediately.

You will find the latest keys in the os-images repo.

  • Ensure the original keys are in your local keychain or cached against the server

    • e.g. ssh -i /path/to/os-images/keys/smc_id_rsa.pub smc@<server>

  • Copy the keys with ssh-copy-id -i /path/to/os-images/keys/smc2_id_rsa.pub smc@<server>

  • smcsh into the server (see SSH to Servers)

  • Delete ~/.ssh/smc_id_rsa.pub

  • Delete the first line from ~/.ssh/authorized_keys, which should correspond to the original key

/var Volume

The /var directory is by default mounted on a different volume than the root partition. However, AWS changed their mounting approach to some dynamic naming, and now it breaks the default /etc/fstab.

You’ll need to fix it manually:

  • Run sudo fdisk -l to establish the true location of the extra volume. It will probably look something like /dev/nvme1n1

  • sudo nano /etc/fstab and change the /var line so it looks like: /dev/nvme1n1 /var ext4 defaults,nofail 0 0

  • Reboot the instance: sudo reboot

Resizing the /var Volume

If you need a larger or smaller size than the instance you are copying, you’ll need to manually alter it. Very briefly, the process is:

  • Stop the server and detach the volume you want to resize (aka Volume X)

  • Create a new volume of the right size (aka Volume Y)

  • Spin up a new temporary server (use the tiniest size you can, or my decanter server which I keep kicking around for this purpose)

  • Attach and mount both volumes; Volume X to /mnt/x and Volume Y to /mnt/y

  • You might need to create a filesystem on the Volume Y. Use mkfs.ext4 /dev/<volumeref>

  • Copy the contents from Volume X to Volume Y with $ cp -apx /mnt/x/* /mnt/y

  • Detach both

  • Attach Volume Y to your original server

  • Restart your original server

You’ll probably need to fix the /etc/fstab again.

Hostname

The hostname is usually squiff and there’s some stuff in the ami which keeps overwriting it if you try and set it naively. You’ll need to step through the following to get it working.

  • sudo hostnamectl set-hostname <HOSTNAME>

  • sudo nano /etc/hosts

  • Add <HOSTNAME> to localhost

    127.0.0.1 localhost <HOSTNAME>
This will likely return the error: sudo: unable to resolve host. You can ignore this.

  • sudo nano /etc/cloud/cloud.cfg

  • Comment out the following lines:

    # - set_hostname
    # - update_hostname

  • sudo nano /etc/cloud/cloud.cfg.d/01_debian_cloud.cfg

  • Change manage_etc_hosts to false:

  • Log out and back in

Install NetData

NetData is our resource visualisation tool for our servers.

Install it manually:

This one can take a while, and you will need to confirm a few times during the process

  • Make a note of the IP address it listens on

  • Run sudo systemctl start netdata

  • Run this command:

sudo mkdir /var/log/nginx/netdata/ && sudo nano /etc/nginx/sites-available/netdata.conf && cd /etc/nginx/sites-enabled/ && sudo ln -s ../sites-available/netdata.conf ./ && sudo nginx -s reload

  • When the editor opens, paste in and edit the following snippet:

server {
    listen          80;
    server_name     netdata.<hostname>.dgcsdev.com;
    ignore_invalid_headers off;

    location / {
        proxy_pass http://<hostname>:<netdata_ip>;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }


    error_log       /var/log/nginx/netdata/error.log;
    access_log      off;
}
Don’t forget to replace <hostname> with your server’s actual hostname and <netdata_ip> with the ip it reported earlier.

Holding Page

  • sudo nano /var/www/html/index.html

  • In the editor paste in the contents from here.

Python 3

As the image is quite old, you’ll likely need to update a few things. Python3 in particular, if you want Ansible to work properly with Docker:

$ sudo apt-get update && \
    sudo apt-get upgrade docker && \
    sudo apt-get install python3 && \
    sudo apt-get install python3-pip && \
    sudo pip install --upgrade pip && \
    pip install docker docker-py && \
    pip3 install docker docker-py
You may get asked to install something to GRUB. Select all devices and if any fail just choose to continue.
If pip3 is not installed by the end of this process, try sudo apt-get remove python3-pip; sudo apt-get install python3-pip; pip3 install docker docker-py

Done

Your server is good to go.