System Architecture

Note: If you’re looking for step-by-step instructions on installing Fugue, check out Hello World, Part 1: Fugue Quick Setup.

Like any operating system, Fugue is installed onto your computer – but not just the computer in front of you. Instead, Fugue is a cloud-native system that installs onto your cloud computer: your infrastructure provider account on Amazon Web Services.

The notion of installing Fugue like an operating system is somewhat metaphorical. In practice, installing Fugue in your account means that a few resources will be created in your AWS account that enable Fugue to run. These include at least one compute instance to host a Conductor, a VPC to host this instance, as well as queues, topics, object and key-value storage, and appropriate security roles and principals.

Installation Scope

Fugue is installed into an Amazon Web Services account. The installation occupies one account, and one region within the account, although it can communicate with any region’s APIs once it is installed. This way, you can install Fugue in a single region, but manage infrastructure in many.

The Fugue Conductor may only be installed in the following regions:

  • us-east-1
  • us-east-2
  • us-west-2
  • eu-west-1
  • us-gov-west-1

Installation Process

Installation of Fugue is only necessary once per account region. A successful installation follows this outline of steps:

  • You install the Fugue Client Tools, which includes the Fugue CLI, according to the directions here.
  • You run fugue install from a properly configured computer with sufficient permissions to install Fugue. The fugue install command blocks at the command line until the rest of the steps are completed.
  • The Fugue CLI uses CloudFormation to “bootstrap” the Fugue installation by creating a stack with the necessary components for the Conductor to boot.
  • As the Conductor boots, it creates (if it does not find) necessary communication and storage resources to run processes and serve requests.
  • Finally, the Conductor sends a “ready” signal back to the installing CLI client to signal success, and then the CLI will tell you Fugue is ready to use.

Fugue supports configuring the Conductor to connect to the internet through a proxy. Refer to Fugue Proxy Support for specifics.

Fugue also supports installing the Conductor into an existing network. Refer to Installing the Conductor in Your Own VPC for specifics.

Major Component Architecture

The major components of Fugue in AWS are:

  • An EC2 instance, hosting the Conductor. We also build a VPC dedicated to hosting Fugue.
  • Several SQS queues and SNS topics, usually prefixed with fugue- (with exceptions). For now, avoid blanket deletion of SQS queues.
  • Several S3 buckets, usually prefixed with fugue- and also containing your account number. The same caveats apply to avoid blanket deletion.
  • Several DynamoDB tables, also usually prefixed with fugue-, but the same caveats apply here as well.
  • Several IAM roles, including those applied to the Conductor to allow it to do process work.

The structural components of the installation are shown here.

                         +
               Local     |     Cloud
                         |
                         |
                         |
                         |
                         |     +---+     +---+
                +-------------->SQS|     |SNS|
                |        |     +-^-+     +-^-+
                |        |       |         |
                |        |       |         |
                |        |       |         |
                |        |       |         |         +---------------+
                |        |       |         |         |               |
   +------------+--+     |     +-+---------+-+       |               |
   |               |     |     |             |       |   AWS APIs    |
   | $> fugue ...  |     |     |  Conductor  +------->   for user    |
   |               |     |     |             |       |   workloads   |
   |   Client PC   |     |     +-+---------+-+       |               |
   |               |     |       |         |         |               |
   +---------------+     |       |         |         +---------------+
  X X X X X X X X X      |       |         |
 X X X X X X X X X       |     +-v--+    +-v-+
XXXXXXXXXXXXXXXXX        |     | S3 |    |DDB|
                         +     +----+    +---+

As you can see, the major components of Fugue are all present in “the cloud,” specifically an infrastructure provider account that you own. Only the Fugue client is present on your local machine. The Fugue Client communicates with the Conductor indirectly, by way of asynchronous messaging. The Conductor has no open ports by default.

On The Client

There are three primary components on the client machine:

  • The Fugue CLI, present in the form of the fugue binary. This is what forms all requests to, and parses all responses from, the Fugue Conductor.
  • The Ludwig compiler, present in the form of the lwc binary. In general, the fugue binary (specifically, in the run and update commands) handles invoking the compiler with the proper switches and preparation, much like make and similar tools.
  • The Fugue Libraries for Ludwig, which you can use to create compositions defining infrastructure. These libraries include both basic building blocks for infrastructure, as well as common infrastructure patterns generalized for reuse and abstracted for simplicity.

On The Conductor

The Conductor uses a service-oriented architecture, with significant application concerns separated among different components. There are many, and an exhaustive list is beyond our scope here, but there are three significant components to know about when considering how the system works (as well as deciphering logs):

  • The Scheduler, which ensures that processes are executed efficiently;
  • The Manager, which coordinates the work of components on the Conductor to plan work, and;
  • The Broker, which coordinates with third-party APIs, such as that of AWS, to execute work.

These components are discussed in additional detail in the next chapter.

The CloudFormation Stack

When you execute fugue install, Fugue deploys a CloudFormation stack in order to install the Conductor in the target AWS account. The CloudFormation stack creates the following resources:

  • Auto Scaling Group
  • DynamoDB Table
  • IAM Role
  • IAM Instance Profile
  • EC2 Route
  • Auto Scaling Group Launch Configuration
  • SNS Topic
  • EC2 Route Table
  • 2 EC2 Subnets
  • 2 EC2 Subnet Route Table Associations
  • EC2 VPC
  • EC2 VPC Gateway
  • EC2 VPC Gateway Attachment
  • EC2 Security Group

Setting the following fields in the conductor block of the fugue.yaml file or in environment variables will affect the CloudFormation stack in different ways:

region
The default region in which the CloudFormation stack is deployed (and where the Conductor is installed) is us-east-1. Specifying another value here changes that region. The Fugue Conductor may only be installed in the following regions: us-east-1, us-east-2, us-west-2, eu-west-1, and us-gov-west-1. (Environment variable: FUGUE_CONDUCTOR_REGION)
ami
Modifying this value changes the Conductor AMI that the CloudFormation stack uses to launch the Conductor EC2 instance. The AMI ID can also be set with fugue init --ami <ami-id> <region>. (Environment variable: FUGUE_CONDUCTOR_AMI)
instanceType
Modifying this value changes the instance type (in <family><generation>.<size> format) used for the Conductor EC2 instance. The default type, m4.large, is the only type Fugue formally supports, so change this value at your own risk. For more information on different instance types, see the AWS documentation. (Environment variable: FUGUE_CONDUCTOR_INSTANCETYPE)
keyName
By default, there are no open ports on the Conductor. However, specifying an SSH keypair name here will change the stack in order to create a security group ingress rule allowing access to the Conductor on port 22 from the IP address where fugue install is run. This enables you to log into the Conductor instance via SSH. Note: the default SSH username is admin. (Environment variable: FUGUE_CONDUCTOR_KEYNAME)
compositionBucket
Specifying a bucket name here means that Fugue stores compiled compositions in the given bucket, rather than creating a new bucket in CloudFormation (named according to the fugue-<account number>-<region> format). Note: Bucket names must be globally unique. (Environment variable: FUGUE_CONDUCTOR_COMPOSITIONBUCKET)
largeValueBucket
Specifying a bucket name here means that Fugue stores large Vars values in the given bucket, rather than creating a new bucket in CloudFormation (named according to the fugue-large-value-<account number>-<region> format). Note: Bucket names must be globally unique. (Environment variable: FUGUE_CONDUCTOR_LARGEVALUEBUCKET)
installAZ
In the region where the CloudFormation stack is deployed, Fugue automatically selects two AZs for the Conductor’s Auto Scaling Group to straddle. You may specify the two availability zones yourself here. Setting your own AZs is useful if you encounter an “AWS Cloudformation stack creation failed” error. (To read more about this error, see the Troubleshooting guide.) (Environment variable: FUGUE_CONDUCTOR_INSTALLAZ)

For more information about fugue.yaml, see the Project Setup Options. To learn general information about how CloudFormation works, see the AWS CloudFormation User Guide.

Installing the Conductor in Your Own VPC

Warning

Only advanced users should attempt to install the Conductor in an existing VPC. If you have questions contact support@fugue.co.

By default, Fugue creates a new VPC with networking components during install and upgrade. However, Fugue supports installing the Conductor in an existing VPC instead of creating a new VPC.

There are three steps:

  1. Set up the network for the Conductor
  2. Set environment variables or fugue.yaml fields
  3. Execute install or upgrade

1. Set up the network for the Conductor

First, set up the network for the Conductor. You must create a VPC and subnets. You can optionally create a security group, and if you don’t, the Conductor will make one for you.

For these steps, we’re configuring the security group rules that go with public subnets we’ve created routes for to reach out to the internet. Depending on your use case, your security group rules may be different:

  • TCP 0.0.0.0/0:443
  • TCP 0.0.0.0/0:53
  • UDP 0.0.0.0/0:53
  • UDP 0.0.0.0/0:123

2. Set environment variables or fugue.yaml fields

Next, set environment variables or fugue.yaml fields to the resource IDs of the network components.

  • vpcId or FUGUE_CONDUCTOR_VPCIDRequired
  • subnetIds or FUGUE_CONDUCTOR_SUBNETIDSRequired
  • securityGroupId or FUGUE_CONDUCTOR_SECURITYGROUPID – Optional

See Project Setup Options for more information.

3. Execute install or upgrade

Finally, execute install and Fugue will set up the Conductor in the existing network. If a Conductor is already running, execute upgrade instead.

Additional Information

  • This feature is not a prerequisite to using the proxy functionality.
  • However, Fugue’s proxy functionality can be used in conjunction with this feature. For example, you might set up the proxy with its own VPC, create a VPC for the Conductor, set up all the connections, and then install Fugue into the VPC you created for the Conductor. Note that Fugue does not provide the connections between the proxy and the Conductor.
  • Your existing network’s infrastructure will not be deleted during uninstall. Fugue only removes the infrastructure it created during the install or upgrade process.

AWS Permissions and the Fugue CLI

The Fugue CLI requires a particular set of AWS permissions. During the install process, Fugue creates two IAM policies, fugue-installer-iam-policy and fugue-user-iam-policy, along with corresponding IAM roles, fugue-installer-<region> and fugue-user-<region>. Together, these policies contain the minimum required permissions to run the Fugue CLI:

  • The installer policy allows only install, upgrade, and uninstall.
  • The user policy allows only account, history, init, kill, property, resume, release, run, status, support, suspend, update, and user, and certain features of policy.

Having separate policies helps promote separation of duties and the security principle of least privilege.