Migrating Infrastructure with Fugue

Table of Contents

• How does migration work?
• Using the import feature
  • How does import work?
  • Supported services for import
• Using Composer and the import feature together
• Example: Bring infrastructure into compliance with Composer and import
• FAQ

How does migration work?

Fugue’s governance, visibility, and control capabilities can be applied to existing cloud resources that were not created by Fugue by migrating those resources to enable Fugue management. Migrating resources to Fugue is a two-step process.

• First, Composer or Transcriber is used to generate a composition that reflects the existing resources in an account.
• Next, the import feature instructs Fugue to begin managing those resources.

Both Composer (via Generate Composition) and Transcriber can scan your AWS account for existing infrastructure and generate a Ludwig composition representing your resources. After using Composer or Transcriber to analyze your AWS accounts and generate the appropriate Ludwig compositions, the import function, executed with fugue run --import, then matches resources between your Ludwig composition and your existing account. Once the process has been successfully created, Fugue treats it like any other process – you can configure enforcement, you can apply validations, you can write policy-as-code around it, you can update or kill the process, and so on.

Note: Ludwig compositions used for migration include resourceId fields which can be created by hand, or generated using Composer or Transcriber – both included in the Fugue Client Tools. In general, we recommend using Composer or Transcriber for this process, but if you have specific questions reach out to support@fugue.co.

Using the import feature

Import (e.g. fugue run <sample.lw> --import) is the second step in the Fugue migration process. It allows you to import and manage existing resources as if they were provisioned by Fugue.

How does import work?

Fugue import matches resource definitions in a user-created or auto-generated Ludwig composition with your existing resources in AWS. An import is activated by passing the --import flag to fugue run, enabling Fugue’s Conductor to match resourceIDs specified within your composition to resources that exist in your account.

For example:

import-this-vpc: EC2.Vpc.new {
  cidrBlock: “10.0.0.0/16”,
  region: AWS.Us-east-2,
  resourceId: “vpc-1234abcd”
}

Supported services for import

Currently supported services include:

Name of service	Supported for import
ASG AutoScaling Groups	aws-autoscaling-autoscaling-groups
ASG AutoScaling Launch Configurations	aws-autoscaling-launch-configurations
ASG AutoScaling Scaling Policies	aws-autoscaling-scaling-policies
CloudWatch Alarms	aws-cloudwatch-alarms
DynamoDB Tables	aws-dynamodb-tables
EC2 Customer Gateways	aws-ec2-customer-gateways
EC2 DHCP Options	aws-ec2-dhcpoptions
EC2 Elastic IPs	aws-ec2-elastic-ip-addressess
EC2 Instances	aws-ec2-instances
EC2 Internet Gateways	aws-ec2-internet-gateways
EC2 Network ACLs	aws-ec2-network-acls
EC2 Network Interfaces	aws-ec2-network-interfaces
EC2 Route Tables	aws-ec2-route-tables
EC2 Security Groups	aws-ec2-security-groups
EC2 Subnets	aws-ec2-subnets
EC2 Volumes	aws-ec2-volumes
EC2 VPC Endpoints	aws-ec2-vpc-endpoints
EC2 VPC Peering	aws-ec2-vpc-peering-connections
EC2 VPCs	aws-ec2-vpcs
EC2 VPN Gateways	aws-ec2-vpn-gateways
ELB Load Balancers	aws-elasticloadbalancing-loadbalancers
ELBv2 Load Balancers	aws-elasticloadbalancing-loadbalancers
ELBv2 Target Groups	aws-elasticloadbalancing-target-groups
IAM Instance Profiles	aws-iam-instance-profiles
IAM Policies	aws-iam-policies
IAM Roles	aws-iam-roles
Lambda Functions	aws-lambda-functions
RDS Instances	aws-rds-instances
RDS Subnet Groups	aws-rds-subnet-groups
S3 Buckets	aws-s3-buckets
SNS Topics	aws-sns-topics
SNS Queues	aws-sqs-queues

More services are forthcoming. Contact support@fugue.co with any questions.

Using Composer and the import feature together

The combination of Composer and the import feature simplify the process of bringing your infrastructure into Fugue. Use Composer’s “Generate Composition” feature to scan the desired account, and it will output a composition that includes the resource IDs of each resource it scans. The composition can then be run with the --import flag in order to import your infrastructure.

Composer and Transcriber may be used interchangeably in the migration process. Transcriber is a command-line tool that scans an AWS account and creates a composition representing the resources it finds. Composer is a composition visualization tool and editor that offers a Generate Composition feature powered by Transcriber.

Best practices

In order to optimize your experience with Composer or Transcriber, and the import feature, consider the following best practices.

• While you can write the migration composition yourself, we recommend using Composer to generate the migration composition. It’s simpler, faster, and more accurate. (Note: Transcriber can be used directly from the command line as well.)
• Instead of migrating all resources within an account into a single Fugue process, we recommend creating multiple Fugue processes during migration. Processes can be used to separate concerns based on your organization’s needs. For example, processes could group resources by type (e.g., network, compute), team, or use case (e.g., dev/test/prod). Transcriber works best when scanning 40 resources or less.
• Tag your resources when possible. Transcriber and Composer can include or exclude resources by tags when scanning an account, which means less time spent processing the output prior to importing it. For example, the -k or --tag-key option filters on a given tag key, and the -p or --tag-pair option filters on a given tag key and value.
• Transcribed compositions should be inspected and processed before migration. Because Composer, the import feature, and Fugue make a very powerful team, it’s best to make sure the transcription doesn’t include anything you don’t want Fugue to manage (such as your IAM login credentials or default VPC).
• Always do a dry-run before migrating. Simply execute fugue run <composition.lw> --dry-run --import. This is a good way to catch certain errors that could impact your migration. Dry-runs are also a best practice for executing other process management commands, such as update, resume, kill, or non-migration run.
• Use Composer to visualize your infrastructure. After you use the Generate Composition feature to create a composition of the resources in your account, you can explore a diagram of the resources and see how they interrelate before you bring them under Fugue’s control.
• Tip: You can import existing infrastructure, modify mutable attributes of the existing infrastructure, and create new infrastructure in the same composition. One fugue run --import handles it all.

Caveats and things to avoid

• Killing a process destroys the process’s infrastructure, including migrated resources. If you use Transcriber or Composer to prepare a composition for import, make sure you remove anything from the transcription that you don’t want Fugue to manage, such as your default VPC with its associated resources, and IAM resources such as login credentials.
  • If you experience a failed run on a migrated process for any reason, don’t kill the process. The safest course of action is to release the process, edit the composition as needed, and then run it again with --import.
• When you use Transcriber or Composer to prepare a migration composition, check for default resources and delete or edit them out of the composition. If you import AWS-created default resources (as opposed to resources you create explicitly), you’ll be unable to kill the process. For example, when you create a VPC, AWS generates a default security group and default route table. Transcriber and Composer will include these defaults in its generated composition, and you can successfully import them, but because AWS doesn’t allow you to delete the default security group or route table, the process will be stuck in a Killing state.
  • Workaround: Suspend the process, release it, and transcribe the infrastructure again, this time excluding the default resource. When you run the composition with --import again, you’ll be able to kill the resulting process.
• When you prepare to import a resource, be sure all of its non-default dependencies are included in the transcribed composition. Say you’ve created a VPC with two subnets via the AWS Management Console – we’ll call them VPC-A, subnet-1, and subnet-2. Let’s also say you’ve transcribed the account, and in the process of editing the generated composition, you’ve accidentally deleted the subnet-2 binding. If you then use the --import flag to run the composition, migrating VPC-A and subnet-1 but not subnet-2, you won’t be able to kill the process. On kill, Fugue will attempt to delete VPC-A but will fail because deleting a resource also deletes its dependencies, and since subnet-2 is a dependency of VPC-A, and Fugue does not manage subnet-2, the process will be stuck in a Killing state.
  • Workaround: Suspend the process, release it, and transcribe the infrastructure again, this time keeping subnet-2. When you run the composition with --import again, you’ll be able to kill the resulting process.
• Don’t forget to use the --import flag when you run a composition whose resources you intend to migrate. If you do a normal fugue run on it, Fugue ignores the resourceId fields and creates all new infrastructure.
  • If this occurs and your composition does not contain idempotent resources (S3, IAM, etc.), the process will successfully run and create a complete copy of your infrastructure. You’ll be able to kill the process.
  • If this occurs and your composition references existing idempotent resources, the process run will fail but you’ll be able to kill it.
• Executing an import run immediately after executing an import dry-run, or executing two import dry-runs in a row, may fail with an error stating that resources are already managed by another process.
  • Workaround: Wait 30 seconds and try again.
• EC2 instances can’t be imported if the Ludwig declaration uses EC2.InstanceBlockDeviceMapping, which is deprecated. Instead, rewrite the instance binding using the new style of specifying EBS volumes on EC2 instances. For more information, see A New Way of Specifying EBS Volumes on EC2 Instances.
  • Tip: Transcriber and Composer use the new style when they transcribe EC2 instances in your account, so if you intend to import a transcribed composition, you’re all set.
• Certain resources do not support external references. When you transcribe an account prior to migration, the generated composition may contain external resources, especially if you exclude some AWS services from transcription. Running such a composition will produce a validation error, whether it’s an --import run or a non-import run. For more information, see Referring to External Resources.

If you are experiencing an issue that isn’t described here, reach out to support@fugue.co.

Use cases

• Use either Composer or Transcriber with the import feature to update mutable properties on migration. If a bad actor gives public access to non-Fugue-managed S3 buckets, you can transcribe the resources and change the bucket policies by adding the policy to the transcribed composition. When you run the composition with the --import flag, the bucket policies will be corrected and the buckets will be under Fugue management.
• Use either Composer or Transcriber with the import feature to separate applications into different processes. If a company has 20 apps in an AWS account and wants to separate them so each app has its own process, and all infrastructure attached to an app is tagged with the app name, you can filter on that tag when running Transcriber and then import the infrastructure for that app. You’ll need to repeat the process for each separate app tag. When you’re done, all apps are contained in separate processes and all app resources are managed by Fugue.
• Use either Composer or Transcriber with the import feature to bring existing infrastructure into compliance with a policy. If you have existing, noncompliant resources in your account, you can transcribe the resources, apply a validation library to the composition, and then edit the Ludwig in accordance with the policy. When you import the infrastructure, the resources will be compliant and managed by Fugue.

In fact, we’re going to demonstrate that last use case in the next section.

Example: Bring infrastructure into compliance with Composer and import

Introduction

We’re going to show you a use case for the import feature by applying validations to existing infrastructure. This example walks you through the process step by step.

Prerequisites

You’ll need to have the Fugue CLI set up and the Conductor installed before you can run this example. If you haven’t done so yet, it just takes a few quick steps.

What we’ll do in this example

Suppose your company has a policy that EC2 instances must receive public DNS hostnames, so all VPCs need to have DNS hostnames enabled.

In order to demonstrate how to bring existing infrastructure into compliance with the corporate policy, we’ll do the following:

1. Use the AWS CLI to create a noncompliant VPC with an accompanying DHCP options set and subnet.
2. Transcribe the new infrastructure.
3. Edit the composition to import the validation that enforces the corporate policy.
4. Modify the composition to bring it into compliance.
5. Run the composition with the --import flag to bring the resources under Fugue management.

What we’ll have when we’re done

Three resources under Fugue management:

• A compliant VPC
• A DHCP options set
• A subnet

How long it will take

About 20 minutes.

Download

You can download the source code for this example from Github:

• CreateVPC.sh
• EnableDNSHostnames.lw

Save it to the directory in which you ran fugue init during the Quick Setup.

Get editor plug-ins here.

Let’s go!

Creating noncompliant infrastructure

We’ve provided you with two files:

• EnableDNSHostnames.lw, a design-time validation that throws an error if a VPC doesn’t have DNS hostnames enabled
• CreateVPC.sh, a shell script to create a VPC, DHCP options set, and subnet

We’re going to focus on the shell script for this part.

The script uses the AWS CLI to create infrastructure according to the region in the $FUGUE_DEMO_REGION environment variable. Pick any region you like, but regions without a lot of other resources work best.

To set the environment variable, we’re going to use the South America (São Paulo) region:

export FUGUE_DEMO_REGION=sa-east-1

Next, we need to make the shell script executable:

chmod 755 CreateVPC.sh

Now you can execute the script like so:

./CreateVPC.sh

You’ll see output like this:

Creating the following resources with the AWS CLI:
vpc-f08a9394
dopt-f7016093
subnet-73e07f14
All done!

Check out the AWS Management Console in whichever region you’ve selected to visually confirm the new resources.

Here in sa-east-1, we see the VPC...

A new VPC in the AWS Management Console.

and the subnet...

A new subnet in the AWS Management Console.

and the DHCP options set.

A new DHCP options set in the AWS Management Console.

Transcribing the infrastructure

For the next step, we’re going to transcribe the resources we just created. This infrastructure is external to Fugue, so transcribing it is the first step to migrating it.

Composer is part of the Fugue Client Tools, so if you haven’t installed it yet, go ahead and install it now.

Open up Composer and select the Generate Composition button. Select your AWS credentials profile or enter your credentials manually and pick your $FUGUE_DEMO_REGION from the dropdown list.

Composer has an option to filter by AWS service, so under Services to Include, select the following services from the Excluded AWS Services list and move them to Included AWS Services:

• aws-ec2-dhcpoptions
• aws-ec2-subnets
• aws-ec2-vpcs

Because it’s a best practice to tag resources, the shell script added the key “Application” and value “FugueImportDemo” to the infrastructure it created. Composer has an option that allows us to filter by tag key, so we’ll enter Application in the text box labeled “Only include resources that match the following AWS tag.”

Settings for Composer’s Generate Composition feature.

Then, select the Generate Composition button at the bottom, where you’ll be prompted to name the composition and save it wherever you’d like. We’ll name ours TranscribedResources.lw.

Composer generates the composition and then displays it side-by-side with a visualization of the infrastructure:

Composer has generated a composition and a visual representation of the infrastructure.

Tip: If you’d prefer to use Transcriber to generate the composition, you can use the following command:

fugue-transcriber --region $FUGUE_DEMO_REGION -i aws-ec2-vpcs -i aws-ec2-dhcpoptions -i aws-ec2-subnets -k Application TranscribedResources.lw

Whether you’ve used Composer or Transcriber, the composition will look something like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: sa-east-1
############################################################
region: AWS.Sa-east-1


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-f7016093
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
  domainName: "sa-east-1.compute.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
  resourceId: "dopt-f7016093",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
}


############################################################
# Subnet
#   Generated from AWS resource: subnet-73e07f14
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "10.0.1.0/24",
  defaultForAz: False,
  mapPublicIpOnLaunch: False,
  resourceId: "subnet-73e07f14",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
  vpc: ec2-vpc-1,
}


############################################################
# VPC
#   Generated from AWS resource: vpc-f08a9394
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: ec2-dhcpoptions-1,
  enableDnsHostnames: False,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-f08a9394",
  tags: [
    AWS.tag("Application", "FugueImportDemo"),
  ],
}

Note that each resource has the resourceId field. This is the most important Ludwig type for a successful migration. This field contains an identifier that uniquely identifies a resource, and is required in order for Fugue to perform the migration.

If you look in the AWS Management Console, you’ll see that Composer (or Transcriber) has accurately transcribed the three resources we created, right down to the last attribute.

Importing the validation

Now’s the time to find the other file you downloaded, EnableDNSHostnames.lw. It’s a simple design-time validation library that raises an error if the composition importing it has a VPC with the enableDnsHostnames attribute set to False. Move the validation library into the same directory as TranscribedResources.lw, if it’s not there already.

Since this represents the new corporate policy, we’d better make sure our infrastructure is compliant.

Editing required! In Composer, import the validation library to TranscribedResources.lw by adding the following statement after the other import statements (around line 5):

import EnableDNSHostnames

Save your work and Composer will recompile the composition. As expected, you’ll get a validation error because enableDnsHostnames is set to False. Composer marks the erroneous line with a red dot and displays the error message next to the visualization:

Composer displays the composition’s validation error.

Here is the full text of the validation error:

ludwig (validation error):
  "/opt/fugue/lib/Fugue/AWS/EC2/Vpc.lw" (line 98, column 12):
  Validations failed:

     98|   let vpc: EC2.Vpc {
     99|             cidrBlock: cidrBlock,
    100|             tags: tags,
    101|             instanceTenancy: instanceTenancy,
    102|             region: region,
    103|             enableDnsSupport: Optional.unpack(True, enableDnsSupport),
    104|             enableDnsHostnames: Optional.unpack(False, enableDnsHostnames),
    105|             dhcpOptions: case dhcpOptions of
    106|                             | None -> Optional(DhcpOptions.defaultForRegionWithTags{region: region, tags: tags})
    107|                             | _    -> dhcpOptions
    108|           }

    - DNS hostnames must be enabled
      (from EnableDNSHostnames.checkDNSHostnames)

  Stack trace:
    In call to new at "TranscribedResources.lw" (line 52, column 12)

(Note: If you are using Transcriber for this example, compile with this command: lwc TranscribedResources.lw)

Editing required! We don’t want our infrastructure to be noncompliant, so we’ll fix the composition by changing the VPC attribute enableDnsHostnames to True (around line 55).

As a bonus, we’re going to change a mutable characteristic in this VPC so it’s easier to see that Fugue is managing it.

Editing required! Change the tag value to "FugueImportDemo-UPDATE" (around line 61).

The new VPC declaration should look something like this:

ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: ec2-dhcpoptions-1,
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-f08a9394",
  tags: [
    AWS.tag("Application", "FugueImportDemo-UPDATE"),
  ],
}

When you save again, you’ll see that the red dot and error message disappear because the composition has compiled successfully.

Note: We’ve checked the composition for compliance using a design-time validation. If you want Fugue to check the composition’s compliance at runtime, you can load the validation library on the Conductor with the command fugue policy validation-add --name checkDNSHostnames EnableDNSHostnames.lw. Once added, runtime validations are applied to any composition the Conductor runs, not just compositions that import the validation library.

Migrating the infrastructure

It’s time to bring the infrastructure under Fugue management, which means running the composition with the –import flag. Doing a test run first is a best practice, so we’ll add a --dry-run flag:

fugue run TranscribedResources.lw --import --dry-run

This helps us catch any errors that might prevent Fugue from successfully importing the infrastructure.

You should see output like this:

[ fugue run ] Dry running /Users/main-user/projects/TranscribedResources.lw...

Asking Fugue to execute a dry run of composition...

FID:   bf3719c6-33d9-4f2c-9d96-e8bf00535443
JobId: '1511228374'

Requests:
  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - subnet-73e07f14
        Tags:
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.71e05c26-a39e-5f47-ac21-bf2f694c98a0
        - Key:   Name
          Value: ec2-subnet-1

  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - dopt-f7016093
        Tags:
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.fb2fc568-d148-5bb8-a75f-206ba53e9e54
        - Key:   Name
          Value: ec2-dhcpoptions-1

  - aws.ec2.modify_vpc_attribute:
      Region: sa-east-1
      Params:
        EnableDnsHostnames:
          Value: true
        VpcId: vpc-f08a9394

  - aws.ec2.create_tags:
      Region: sa-east-1
      Params:
        Resources:
        - vpc-f08a9394
        Tags:
        - Key:   Application
          Value: FugueImportDemo-UPDATE
        - Key:   Fugue ID
          Value: bf3719c6-33d9-4f2c-9d96-e8bf00535443.cbaa1722-2b8b-5aee-b288-b0ef725f065d
        - Key:   Name
          Value: ec2-vpc-1

The output includes a list of AWS API calls that Fugue would make to import each resource. As you can see, there’s a call to aws.ec2.modify_vpc_attribute, which represents the change we made to enable DNS hostnames. There’s also a number of aws.ec2.create_tags calls, which is because when Fugue creates or imports a resource it can tag, it adds a tag with "Fugue ID" as the key and a string representing the Fugue ID as the value.

All looks good, so it’s finally time to import the resources:

fugue run TranscribedResources.lw --import -a importDemo

You should see the usual run output, indicating the CLI has sent the request to the Conductor.

Wait a little bit and then check the status of the process:

fugue status

When the run has successfully completed and has migrated the infrastructure to Fugue, you’ll see status output like this:

Fugue Status Report for user/xxxxxxxxxxxx - Mon Nov 20 2017 8:41pm

State    Updated    Created    Account              FID                                   Alias       Flags    Last Message    Next Command
-------  ---------  ---------  -------------------  ------------------------------------  ----------  -------  --------------  --------------
Running  8:40pm     8:40pm     fugue-1507311378131  63284a23-9b14-4625-a73c-7dd522f581ae  importDemo  -e       SUCCEEDED

To visually confirm that Fugue is managing the infrastructure, visit the VPC Management Console and click on the VPC the shell script created.

You should see some changes:

• The VPC has been named ec2-vpc-1
• Its DNS hostnames setting says “yes”
• Its tags include the new Fugue ID
• Its Application tag has an updated value, “FugueImportDemo-UPDATE”

We imported this VPC and enabled DNS hostnames.

The imported VPC has newly updated tags.

It’s official – we just brought existing infrastructure into compliance with a corporate policy and enabled it to be managed by Fugue! Since it’s under Fugue’s control now, we can treat the process as we would any other Fugue process. We can get drift notifications, we can update the process, we can apply design-time or runtime validations to it, we can interact with it via the Fugue API Server, we can kill it.

Killing the running process

We’ve finished the demonstration, so we’ll terminate the process and destroy the resources we’ve created:

fugue kill importDemo

When fugue status returns this message, you’re all done:

Fugue Status Report for <user>/<account> - Fri Mar 17 2017 5:31pm


State    Updated    Created    Account    FID    Alias    Flags    Last Message    Next Command
-------  ---------  ---------  ---------  -----  -------  -------  --------------  --------------
Nothing to see here. Go create something! :-)

FAQ

What information does my composition need to contain to successfully import?

Before you run import you must have a valid composition with the required resource definitions and resourceIDs.

Note: If a resourceID isn’t present for every resource you are attempting to import, Fugue will create a new version of that resource, which is, in general, not the desired behavior. To avoid this scenario we recommend using Composer or Transcriber to generate the Ludwig for your composition.

What services can I import?

See the list of supported services on this page - here. For details on supported services using Transcriber - refer here.

What is a resourceId?

The resourceId is an identifier used to uniquely identify a resource in an AWS account. Where possible, the resourceId is the ARN, but many resources do not have ARNs, in which case the resourceId is some other unique identifier associated with the resource.

Note: Due to the difficulty of determining the resourceId for a given resource, we recommend using Composer or Transcriber to generate Ludwig compositions to import, rather than writing them by hand.

What determines which services I have permission to import?

Composer and Transcriber simply require read-only access. For import the AWS profile/role associated with your Fugue account will need permission to view and modify any resources you want to import. Click here to read more about permissions.

What is the relationship between import and Composer/Transcriber?

Composer or Transcriber analyzes AWS accounts and generates Ludwig compositions that can be used to import the resources in those accounts. Import references the Ludwig compositions and brings the infrastructure into Fugue as processes for management.

Do you have an example of how the migration process works?

Yes we do, the example starts here.

What if I have comments or questions?

You can reach out to us at support@fugue.co.