System Architecture

Note: If you’re looking for step-by-step instructions on installing Fugue, check out Hello World, Part 1: Fugue Quick Setup.

Like any operating system, Fugue is installed onto your computer – but not just the computer in front of you. Instead, Fugue is a cloud-native system that installs onto your cloud computer: your infrastructure provider account on Amazon Web Services.

The notion of installing Fugue like an operating system is somewhat metaphorical. In practice, installing Fugue in your account means that a few resources will be created in your AWS account that enable Fugue to run. These include at least one compute instance to host a Conductor, a VPC to host this instance, as well as queues, topics, object and key-value storage, and appropriate security roles and principals.

Installation Scope

Fugue is installed into an Amazon Web Services account. The installation occupies one account, and one region within the account, although it can communicate with any region’s APIs once it is installed. This way, you can install Fugue in a single region, but manage infrastructure in many.

The Fugue Conductor may only be installed in the following regions:

  • us-east-1
  • us-east-2
  • us-west-2
  • eu-west-1
  • us-gov-west-1

Installation Process

Installation of Fugue is only necessary once per account region. A successful installation follows this outline of steps:

  • You install the Fugue Client Tools, which includes the Fugue CLI, according to the directions here.
  • You run fugue install from a properly configured computer with sufficient permissions to install Fugue. The fugue install command blocks at the command line until the rest of the steps are completed.
  • The Fugue CLI uses CloudFormation to “bootstrap” the Fugue installation by creating a stack with the necessary components for the Conductor to boot.
  • As the Conductor boots, it creates (if it does not find) necessary communication and storage resources to run processes and serve requests.
  • Finally, the Conductor sends a “ready” signal back to the installing CLI client to signal success, and then the CLI will tell you Fugue is ready to use.

Major Component Architecture

The major components of Fugue in AWS are:

  • An EC2 instance, hosting the Conductor. We also build a VPC dedicated to hosting Fugue.
  • Several SQS queues and SNS topics, usually prefixed with fugue- (with exceptions). For now, avoid blanket deletion of SQS queues.
  • Several S3 buckets, usually prefixed with fugue- and also containing your account number. The same caveats apply to avoid blanket deletion.
  • Several DynamoDB tables, also usually prefixed with fugue-, but the same caveats apply here as well.
  • Several IAM roles, including those applied to the Conductor to allow it to do process work.

The structural components of the installation are shown here.

               Local     |     Cloud
                         |     +---+     +---+
                +-------------->SQS|     |SNS|
                |        |     +-^-+     +-^-+
                |        |       |         |
                |        |       |         |
                |        |       |         |
                |        |       |         |         +---------------+
                |        |       |         |         |               |
   +------------+--+     |     +-+---------+-+       |               |
   |               |     |     |             |       |   AWS APIs    |
   | $> fugue ...  |     |     |  Conductor  +------->   for user    |
   |               |     |     |             |       |   workloads   |
   |   Client PC   |     |     +-+---------+-+       |               |
   |               |     |       |         |         |               |
   +---------------+     |       |         |         +---------------+
  X X X X X X X X X      |       |         |
 X X X X X X X X X       |     +-v--+    +-v-+
XXXXXXXXXXXXXXXXX        |     | S3 |    |DDB|
                         +     +----+    +---+

As you can see, the major components of Fugue are all present in “the cloud,” specifically an infrastructure provider account that you own. Only the Fugue client is present on your local machine. The Fugue Client communicates with the Conductor indirectly, by way of asynchronous messaging. The Conductor has no open ports by default.

On The Client

There are three primary components on the client machine:

  • The Fugue CLI, present in the form of the fugue binary. This is what forms all requests to, and parses all responses from, the Fugue Conductor.
  • The Ludwig compiler, present in the form of the lwc binary. In general, the fugue binary (specifically, in the run and update commands) handles invoking the compiler with the proper switches and preparation, much like make and similar tools.
  • The Fugue Libraries for Ludwig, which you can use to create compositions defining infrastructure. These libraries include both basic building blocks for infrastructure, as well as common infrastructure patterns generalized for reuse and abstracted for simplicity.

On The Conductor

The Conductor uses a service-oriented architecture, with significant application concerns separated among different components. There are many, and an exhaustive list is beyond our scope here, but there are three significant components to know about when considering how the system works (as well as deciphering logs):

  • The Scheduler, which ensures that processes are executed efficiently;
  • The Manager, which coordinates the work of components on the Conductor to plan work, and;
  • The Broker, which coordinates with third-party APIs, such as that of AWS, to execute work.

These components are discussed in additional detail in the next chapter.

The CloudFormation Stack

When you execute fugue install, Fugue deploys a CloudFormation stack in order to install the Conductor in the target AWS account. The CloudFormation stack creates the following resources:

  • Auto Scaling Group
  • DynamoDB Table
  • IAM Role
  • IAM Instance Profile
  • EC2 Route
  • Auto Scaling Group Launch Configuration
  • SNS Topic
  • EC2 Route Table
  • 2 EC2 Subnets
  • 2 EC2 Subnet Route Table Associations
  • EC2 VPC
  • EC2 VPC Gateway
  • EC2 VPC Gateway Attachment
  • EC2 Security Group

Setting the following fields in the conductor block of the fugue.yaml file will affect the CloudFormation stack in different ways:

The default region in which the CloudFormation stack is deployed (and where the Conductor is installed) is us-east-1. Specifying another value here changes that region. The Fugue Conductor may only be installed in the following regions: us-east-1, us-east-2, us-west-2, eu-west-1, and us-gov-west-1.
Modifying this value changes the Conductor AMI that the CloudFormation stack uses to launch the Conductor EC2 instance. The AMI ID can also be set with fugue init --ami <ami-id> <region>.
Modifying this value changes the instance type (in <family><generation>.<size> format) used for the Conductor EC2 instance. The default type, m4.large, is the only type Fugue formally supports, so change this value at your own risk. For more information on different instance types, see the AWS documentation.
By default, there are no open ports on the Conductor. However, specifying an SSH keypair name here will change the stack in order to create a security group ingress rule allowing access to the Conductor on port 22 from the IP address where fugue install is run. This enables you to log into the Conductor instance via SSH.
Specifying a bucket name here means that Fugue stores compiled compositions in the given bucket, rather than creating a new bucket in CloudFormation (named according to the fugue-<account number>-<region> format). Note: Bucket names must be globally unique.
Specifying a bucket name here means that Fugue stores large Vars values in the given bucket, rather than creating a new bucket in CloudFormation (named according to the fugue-large-value-<account number>-<region> format). Note: Bucket names must be globally unique.
In the region where the CloudFormation stack is deployed, Fugue automatically selects two AZs for the Conductor’s Auto Scaling Group to straddle. You may specify the two availability zones yourself here. Setting your own AZs is useful if you encounter an “AWS Cloudformation stack creation failed” error. (To read more about this error, see the Troubleshooting guide.)

For more information about fugue.yaml, see the line-by-line documentation. To learn general information about how CloudFormation works, see the AWS CloudFormation User Guide.

AWS Permissions and the Fugue CLI

The Fugue CLI requires a particular set of AWS permissions. During the install process, Fugue creates two IAM policies, fugue-installer-iam-policy and fugue-user-iam-policy, along with corresponding IAM roles, fugue-installer-<region> and fugue-user-<region>. Together, these policies contain the minimum required permissions to run the Fugue CLI:

  • The installer policy allows only install, upgrade, and uninstall.
  • The user policy allows only account, history, init, kill, property, resume, release, run, status, support, suspend, update, and user, and certain features of policy.

Having separate policies helps promote separation of duties and the security principle of least privilege.