Building Compute: Instances

Lots of computing infrastructure relies upon compute instances, which are a lot like virtual machines but meant to be used a bit differently. Compute instances can serve as long-lasting servers, analogous to rack-mounted servers, but they can just as easily serve as ephemeral, disposable computation power that is only used for a little while and then discarded. This example shows you how to launch instances with Fugue.

Overview

Prerequisites

Ideally, you’ll have worked through the Build A Network example. You can use your own VPC or subnets, but remember that VPCs are free, and this example is written in such a way as to expect the network built in that example.

What We’ll Do In This Example

We’ll cover how to build a mock jumpbox that can be used for making SSH connections to other instances for maintenance work.

What We’ll Have When We’re Done

An EC2 instance, along with requisite security groups and the like.

How Long It Will Take

About 15 minutes.

Download

You can download the source code for this example here:

NetworkLib.lw
The library we are using to launch instances, which defines our network based on the Build A Network example.
Compute-1.lw
The composition we’re building in this example, which defines a bunch of compute infrastructure.

Get editor plug-ins here.

Let’s Go!

Importing the Library

The composition keyword cannot be present in an imported module or library.
composition

import # stuff for networks...

We are planning to declare some instances, but in order to do so, we are going to import a library based on the network we declared in the Build A Network example. This way, we’ll have a network to launch instances in. In order to import Network.lw, we’ve removed the composition keyword from the top of the file and saved it as NetworkLib.lw. We also made a couple other modifications when we turned the composition into a library, but we’ll get to those shortly.

In Ludwig, we use the composition keyword to distinguish runnable Ludwig from importable Ludwig (in other words, libraries). You can more or less think of composition as an entry point in other languages, such as a main() function. A program in Ludwig can only have one entry point, therefore an imported module cannot be a composition.

Structuring Compositions

A best-practice for structuring compositions is to create a project or application-specific composition file, and then define infrastructure pertaining to separate concerns in other modules. For example, consider this file tree:

.
├── Lib
│   ├── App.lw
│   └── Vpc.lw
└── RetailSite.lw

In this case, RetailSite.lw is the composition, and Lib/Vpc.lw and Lib/App.lw contain specific concerns of infrastructure pertaining to networks and application compute & storage, respectively. The RetailSite.lw file just ties the two together with a couple of import statements, as well as make the whole collection runnable, like this:

composition

import Lib.Vpc as Intranet
import Lib.App as Site

If you open up the NetworkLib.lw library you downloaded, you may notice a couple other key changes. First of all, there’s a new section near the top of the file:

resources: (
  example-vpc,
  public-10-0-1-0,
  public-10-0-2-0,
  example-igw,
  public-route,
  public-route-table,
  example-elb-sg,
  example-app-sg
)

resources is a top-level tuple that collects all of the resources declared in the original composition under one name. Having these resources together in a single binding will make things simpler when we move ahead to Compute-1.lw and need to build them.

The other thing we changed when we converted Network.lw to NetworkLib.lw was to remove the inet-HTTP and inet-HTTPS security groups, because we won’t need them for Compute-1.lw. Instead, we’ll create a new security group allowing access over SSH.

Creating a New Composition

As in the first example, we must declare that our file is a composition (using the composition keyword), and thus an entry point. We’ll also import the base AWS library, as well as the EC2 library for declaring compute resources.

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import NetworkLib as Net

In addition to importing the NetworkLib library, we have to explicitly bring all of its bits and pieces into scope of the current composition.

network: Net.resources

Remember that resources binding we mentioned earlier? Since all resources must be in a top-level binding of the composition to be built, we use this top-level binding in the composition to reference the resources top-level binding in the library. If we didn’t include this line at all, only the explicitly named resources in the composition would be built, giving us only the bare essentials required to launch the instance and leaving us without an internet gateway and other crucial components declared in the library. The composition would compile and the process would run, but the instance would be inaccessible. So, we’ll refer to the resources we need from the library via the Net alias for the NetworkLib.

Making an Instance

Oh boy! The time has come to make an instance. Single instances are good for lots of things, but in our case this is going to be an SSH jumpbox – or at least, nominally; in practice, we’ll just launch a plain Amazon Linux server.

jumpbox: EC2.Instance.new {
  instanceType: EC2.T2_micro,
  subnet: Net.public-10-0-1-0,
  image: "ami-7172b611",
  keyName: "admin-login",
  securityGroups: [inet-SSH],
  monitoring: False
}

If you’ve launched an instance on AWS before, this is probably a familiar sight. We’ve established an instance type, and t2.micro instance is plenty for a server that is mostly idle and only needs to provide sshd. We’ve placed it in one of our public subnets, and from that it will inherit placement in an availability zone, VPC, and region. The image field declares which Amazon Machine Image (AMI) to use for the basis of the instance (all instances are based on some AMI). We’re planning to use the server for SSH, so of course we have established that it should use the SSH key admin-login. (More on that in a moment!) Finally, we placed the instance in an inet-SSH security group. We haven’t made that security group yet, but that’s next.

Creating the Right Security Group

For our jumpbox to be effective, we’ll need to be able to initiate connections to it on TCP port 22 from internet IP addresses. This definition of such a security group should be familiar from earlier examples.

inet-SSH: EC2.SecurityGroup.new {
  description: "Allow SSH traffic from the Internet",
  ipPermissions: [
    EC2.IpPermission.ssh(EC2.IpPermission.Target.all)
    # Note this ^ is a bad idea for many applications. Use at your own risk.
  ],
  ipPermissionsEgress: None,
  vpc: Net.example-vpc
}

This creates a new security group that allows ingress on TCP port 22 from the internet. Now, anyone with the right SSH key can log in to hosts in this group via SSH.

Adding the SSH Key

Fugue automates a lot of work, but there are some things that must be done manually (for now). A great example of this is adding SSH keys to your AWS account. For the next step to work, you have to create the admin-login key in your account. You can read more about how to do this here.

Starting the Fugue process

We can launch this instance (as well as the network resources it depends on) by simply running the Compute-1.lw composition. You can accomplish this by running:

fugue run Compute-1.lw -a example-compute-1

You can see the Build A Network example or the CLI Reference for more information on the fugue run command.

Warning

If you forgot to create the admin-login SSH key before executing fugue run, the process will eventually encounter an error. Here’s how to fix it.

  • If you execute fugue status and the process does not report an ERROR state, you can go create the key in your AWS account right away, and Fugue will find the key.
  • If you execute fugue status and the process does report an ERROR state, you can run fugue suspend to temporarily halt the process, then run fugue resume to resume it, and Fugue will look for the key again.

Connecting to the Jumpbox

To prove that we’ve really built something, we should connect to it and test it. Proof of a working jumpbox is being able to SSH to it, so we’ll do that now.

You may have made an SSH connection to a server on AWS before. It looks like this:

$ ssh -i ssh-key-name.pem username@domain-or-ip

We know the key name; admin-login.pem. We know the user name, since this is Amazon Linux; ec2-user. What we don’t know, yet, is the domain name or IP address for the instance.

Fortunately, the status command can get us this information. In fact, it can get us a lot of information. If you try running fugue status with a FID or alias argument, you’ll see quite a bit of data. It’s all JSON, so you can use jq or other utilities to process this information for automation and scripting purposes. For our purposes, a quick grep gets us the IP address we’re missing for SSH.

$ fugue status example-compute-1 | grep PublicIpAddress
                    "PublicIpAddress": "52.42.37.245",

Now, we know how to connect to our jumpbox.

$ ssh -i admin-login.pem ec2-user@52.42.37.245
The authenticity of host '52.42.37.245 (52.42.37.245)' can't be established.
ECDSA key fingerprint is SHA256:YgIVrbCE/bf6UQN+gEN2QHYfHshpS7ot09q/7aEVC08.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '52.42.37.245' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
8 package(s) needed for security, out of 17 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-10-0-1-70 ~]$

Cool! From here, our jumpbox can be used to access things in our VPC via private IP addresses, and other useful admin work.

Killing the Fugue process

As in previous examples, we don’t want our example infrastructure laying around gathering dust, so we’ll want to clean it up by killing the process.

Next Steps

We’ll cover these in future examples, but you can start investigating AutoScaling Groups and Elastic Load Balancers and adding them to your VPC. Or, if you’re interested in setting up SSH jumpboxes or bastion hosts, you can read more here.