Fugue Transcriber

Fugue Transcriber (currently in beta) is a command-line tool that scans your AWS account for existing infrastructure and generates a Ludwig composition representing your resources.

Transcriber is packaged with the Fugue Client Tools. Create a Fugue account, if you haven’t already, to download the Fugue Client Tools from the Download Portal. Find installation instructions at the Fugue Quick Setup.

Note: If you prefer to work in a GUI, Fugue Composer has a “Generate Composition” feature that offers the same major options as the Transcriber command-line tool. Find more information at Fugue Composer.

Usage

fugue-transcriber [OPTIONS] OUT

Options

--region TEXT
Region value to substitute for AWS::Region.
-k | --tag-key TEXT
Limit scan to AWS resources that have this tag key. Use one of --tag-key or --tag-pair.
-p | --tag-pair TEXT...
Two arguments to limit scan to AWS resources that have this tag key and value. Example: --tag-pair Name my-vpc. Use one of --tag-key or --tag-pair.
-i | --include-service TEXT
Include only this service when querying the cloud. May be used more than once.
-x | --exclude-service TEXT
Exclude this service when querying the cloud. May be used more than once.
--include-fugue
Include AWS resources tagged with a Fugue ID.
-q | --quiet
Quiet mode
--debug
Debug mode
-v | --version
Print the current version number and exit.
-l | --list-services
List services covered by transcriber and exit.
-h | --help
Show this message and exit.

Definition

Transcriber generates Ludwig compositions from AWS account resources. Each resource declaration in the composition is preceded by a comment that indicates the type and ID of the resource that it represents.

Transcriber output is sent to OUT. For OUT, use a filename to save to that file or - to send results to standard output.

To view available services and for a list of valid services for the --include-service and --exclude-service options, use the -l or --list-services option.

Supported Services

Currently supported services include:

Name of service Usage in Transcriber
EC2 DHCP Options aws-ec2-dhcpoptions
EC2 Subnets aws-ec2-subnets
EC2 VPCs aws-ec2-vpcs
EC2 Security Groups aws-ec2-security-groups
EC2 Route Tables aws-ec2-route-tables
EC2 Internet Gateways aws-ec2-internet-gateways
EC2 Instances aws-ec2-instances
ASG Launch Configurations aws-autoscaling-launch-configurations
ASG Autoscaling Groups aws-autoscaling-autoscaling-groups
IAM Roles aws-iam-roles
IAM Instance Profiles aws-iam-instance-profiles
IAM Policies aws-iam-policies
DynamoDB Tables aws-dynamodb-tables
ELB Load Balancers aws-elb-loadbalancers
SQS Queues aws-sqs-queues
SNS Topics aws-sns-topics
SNS Subscriptions aws-sns-subscriptions
S3 Buckets aws-s3-buckets
RDS Instances aws-rds-instances
Lambda Functions aws-lambda-functions
CloudFormation Stacks aws-cloudformation-stacks
RDS Option Groups aws-rds-option-groups
RDS Parameter Groups aws-rds-parameter-groups
RDS Subnet Groups aws-rds-subnet-groups
RDS Cluster Parameter Groups aws-rds-cluster-parameter-groups
RDS Clusters aws-rds-clusters
RDS Instances aws-rds-instances
RDS Instance Read Replicas aws-rds-instance-read-replicas

More services are forthcoming.

Transcriber and AWS credentials

Transcriber scans the AWS account associated with the values of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, if those environment variables are set. Otherwise, Transcriber scans the account associated with the default profile of the AWS CLI credentials file (generally located at ~/.aws/credentials).

To have Transcriber scan an account associated with a different profile, export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables associated with the desired account.

Or, make another profile the default account by setting the AWS_DEFAULT_PROFILE environment variable:

$ export AWS_DEFAULT_PROFILE=user2

Examples

Scanning a region other than us-east-1

Transcriber scans the us-east-1 region by default. If you’d like Transcriber to scan a different region, use the --region option. The example below scans the us-west-2 region and saves the output to composition.lw:

fugue-transcriber --region us-west-2 composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-west-2 region.
  Scanning DHCP Options... found 7
  Scanning VPCs... found 2
  Scanning Subnets... found 4
  Scanning Security Groups... found 6
  Scanning Route Tables... found 3
  Scanning Internet Gateways... found 2
  Scanning Instances... found 0
  Scanning Launch Configurations... found 0
  Scanning AutoScaling Groups... found 0
  Scanning IAM Roles... found 12
  Scanning IAM Instance Profiles... found 7
  Scanning IAM Policies... found 12
  Scanning DynamoDB Tables... found 0
  Scanning Load Balancers... found 0
  Scanning SQS Queues... found 0
  Scanning SNS Topics... found 0
  Scanning SNS Subscriptions... found 0
  Scanning S3 Buckets... found 6
  Scanning Lambda Functions... found 0
  Scanning CloudFormation Stacks... found 0
  Scanning RDS option groups... found 0
  Scanning RDS parameter groups... found 0
  Scanning RDS subnet groups... found 0
  Scanning RDS cluster parameter groups... found 0
  Scanning RDS clusters... found 0
  Scanning RDS instances... found 0
  Scanning RDS instance read replicas... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3


############################################################
# Region
#   From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-175bb772
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
  domainName: "us-west-2.compute.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
}


############################################################
# VPC
#   Generated from AWS resource: vpc-922891f7
############################################################
ec2-vpc-2: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: ec2-dhcpoptions-1,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
}

[output trimmed for length]

(Output trimmed for length.)

Limiting results to resources with a specific tag key

If you want Transcriber to generate Ludwig for only resources tagged with a particular key, use the --tag-key or -k option. Transcriber uses this option to filter the results of the services scan. The example below searches for the tag key my-vpc and uses - to print to standard output:

fugue-transcriber --tag-key my-vpc -

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Scanning DHCP Options... found 0
  Scanning VPCs... found 1
  Scanning Subnets... found 0
  Scanning Security Groups... found 0
  Scanning Route Tables... found 0
  Scanning Internet Gateways... found 0
  Scanning Instances... found 0
  Scanning Launch Configurations... found 0
  Scanning AutoScaling Groups... found 0
  Scanning IAM Roles... found 0
  Scanning IAM Instance Profiles... found 0
  Scanning IAM Policies... found 0
  Scanning DynamoDB Tables... found 0
  Scanning Load Balancers... found 0
  Scanning SQS Queues... found 0
  Scanning SNS Topics... found 0
  Scanning SNS Subscriptions... found 0
  Scanning S3 Buckets... found 0
  Scanning Lambda Functions... found 0
  Scanning CloudFormation Stacks... found 0
  Scanning RDS option groups... found 0
  Scanning RDS parameter groups... found 0
  Scanning RDS subnet groups... found 0
  Scanning RDS cluster parameter groups... found 0
  Scanning RDS clusters... found 0
  Scanning RDS instances... found 0
  Scanning RDS instance read replicas... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-241e4e5d
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  tags: [
    AWS.tag("Name", ""),
    AWS.tag("my-vpc", ""),
  ],
}


------------------------------------------------------------
Transcriber complete.

Limiting results to resources with a specific tag key-value pair

If you want Transcriber to generate Ludwig for only resources tagged with a particular key-value pair, use the --tag-pair or -p option. Transcriber uses this option to filter the results of the services scan. The example below searches for the tag key Name and tag value my-subnet, and uses - to print to standard output:

fugue-transcriber --tag-pair Name my-subnet -

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Scanning DHCP Options... found 0
  Scanning VPCs... found 0
  Scanning Subnets... found 1
  Scanning Security Groups... found 0
  Scanning Route Tables... found 0
  Scanning Internet Gateways... found 0
  Scanning Instances... found 0
  Scanning Launch Configurations... found 0
  Scanning AutoScaling Groups... found 0
  Scanning IAM Roles... found 0
  Scanning IAM Instance Profiles... found 0
  Scanning IAM Policies... found 0
  Scanning DynamoDB Tables... found 0
  Scanning Load Balancers... found 0
  Scanning SQS Queues... found 0
  Scanning SNS Topics... found 0
  Scanning SNS Subscriptions... found 0
  Scanning S3 Buckets... found 0
  Scanning Lambda Functions... found 0
  Scanning CloudFormation Stacks... found 0
  Scanning RDS option groups... found 0
  Scanning RDS parameter groups... found 0
  Scanning RDS subnet groups... found 0
  Scanning RDS cluster parameter groups... found 0
  Scanning RDS clusters... found 0
  Scanning RDS instances... found 0
  Scanning RDS instance read replicas... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# Subnet
#   Generated from AWS resource: subnet-38c0bd62
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "10.0.0.0/24",
  defaultForAz: False,
  mapPublicIpOnLaunch: False,
  tags: [
    AWS.tag("Name", "my-subnet"),
  ],
  vpc: EC2.Vpc.external("vpc-241e4e5d", region),
}


------------------------------------------------------------
Transcriber complete.

Including only specific services in the results

To scan only certain AWS services, use the --include-service or -i option. It may be used multiple times in order to include multiple services. The following example limits Transcriber’s results to VPCs and subnets and saves the output to composition.lw:

fugue-transcriber -i aws-ec2-vpcs -i aws-ec2-subnets composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Scanning VPCs... found 3
  Scanning Subnets... found 9

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-c2511fbb
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  tags: [
    AWS.tag("Name", "Fugue Conductor VPC"),
    AWS.tag("aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/fugue/2ca861c0-7e9c-11e7-a103-503acac5c0fd"),
    AWS.tag("aws:cloudformation:logical-id", "FugueVpc"),
    AWS.tag("aws:cloudformation:stack-name", "fugue"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-3: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
}


############################################################
# Subnet
#   Generated from AWS resource: subnet-40951d19
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "172.31.16.0/20",
  defaultForAz: True,
  mapPublicIpOnLaunch: True,
  vpc: ec2-vpc-3,
}

[output trimmed for length]

(Output trimmed for length.)

For a list of supported services and how to refer to them in Transcriber, see Supported Services.

Excluding specific services from the results

To exclude certain AWS services from Transcriber’s scan, use the --exclude-service or -x option. It may be used multiple times to exclude multiple services. The following example excludes VPCs and subnets from the scan results and saves the output to composition.lw:

fugue-transcriber -x aws-ec2-vpcs -x aws-ec2-subnets composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Scanning DHCP Options... found 1
  Scanning Security Groups... found 11
  Scanning Route Tables... found 4
  Scanning Internet Gateways... found 2
  Scanning Instances... found 0
  Scanning Launch Configurations... found 1
  Scanning AutoScaling Groups... found 1
  Scanning IAM Roles... found 12
  Scanning IAM Instance Profiles... found 7
  Scanning IAM Policies... found 12
  Scanning DynamoDB Tables... found 4
  Scanning Load Balancers... found 0
  Scanning SQS Queues... found 16
  Scanning SNS Topics... found 14
  Scanning SNS Subscriptions... found 11
  Scanning S3 Buckets... found 6
  Scanning Lambda Functions... found 0
  Scanning CloudFormation Stacks... found 1
  Scanning RDS option groups... found 0
  Scanning RDS parameter groups... found 0
  Scanning RDS subnet groups... found 0
  Scanning RDS cluster parameter groups... found 0
  Scanning RDS clusters... found 0
  Scanning RDS instances... found 0
  Scanning RDS instance read replicas... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.AutoScaling as AutoScaling
import Fugue.AWS.Cloudformation as Cloudformation
import Fugue.AWS.DynamoDB as DynamoDB
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3
import Fugue.AWS.SNS as SNS
import Fugue.AWS.SQS as SQS


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-ec03ef89
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
  domainName: "ec2.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
}


############################################################
# Security Group
#   Generated from AWS resource: sg-064d9f78
############################################################
ec2-securitygroup-1: EC2.SecurityGroup.new {
  description: "launch-wizard-4 created 2017-05-04T16:10:09.912-04:00",
  ipPermissions: [
    EC2.IpPermission.new {
      fromPort: 22,
      ipProtocol: "tcp",
      target: EC2.IpPermission.Target.ipRanges([
          "173.73.218.131/32",
        ]),
      toPort: 22,
    },
  ],
  ipPermissionsEgress: [
    EC2.IpPermission.new {
      fromPort: 0,
      ipProtocol: "-1",
      target: EC2.IpPermission.Target.all,
      toPort: 0,
    },
  ],
  vpc: EC2.Vpc.external("vpc-76f2ab13", region),
}

[output trimmed for length]

(Output trimmed for length.)

For a list of supported services and how to refer to them in Transcriber, see Supported Services.

Include resources tagged with a Fugue ID

When you fugue run a composition, Fugue tags the resources it creates with a Fugue ID, or FID. (For example, 7e3d8b03-480d-41ba-b430-5dda1f91e538.) By default, Transcriber ignores AWS resources tagged with a FID. If you’d like Transcriber to include these resources in its scan, use the --include-fugue option. Transcriber uses this option to filter the results of the services scan. The following example includes FID-tagged resources and, for simplicity, only scans VPCs. The - directs output to standard out.

fugue-transcriber --include-fugue --include-service aws-ec2-vpcs -

Fugue happens to be running a process that created a VPC tagged my-updated-vpc, so that VPC is included in the results:

Transcriber starting...
Scanning in us-east-1 region.
  Scanning VPCs... found 4

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-65c4911c
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-4d1c292b", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  tags: [
    AWS.tag("Name", "my-updated-vpc"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-c2511fbb
############################################################
ec2-vpc-2: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  tags: [
    AWS.tag("Name", "Fugue Conductor VPC"),
    AWS.tag("aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/fugue/2ca861c0-7e9c-11e7-a103-503acac5c0fd"),
    AWS.tag("aws:cloudformation:logical-id", "FugueVpc"),
    AWS.tag("aws:cloudformation:stack-name", "fugue"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-4: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
}

[output trimmed for length]

------------------------------------------------------------
Transcriber complete.

(Output trimmed for length.)

If you then run the same Transcriber command without --include-fugue, you’ll see that my-updated-vpc is not included:

fugue-transcriber --include-service aws-ec2-vpcs -
Transcriber starting...
Scanning in us-east-1 region.
  Scanning VPCs... found 4

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-c2511fbb
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  tags: [
    AWS.tag("Name", "Fugue Conductor VPC"),
    AWS.tag("aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/fugue/2ca861c0-7e9c-11e7-a103-503acac5c0fd"),
    AWS.tag("aws:cloudformation:logical-id", "FugueVpc"),
    AWS.tag("aws:cloudformation:stack-name", "fugue"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-3: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
}

[output trimmed for length]

------------------------------------------------------------
Transcriber complete.

(Output trimmed for length.)

Note: --include-fugue only applies to resources tagged with a FID – resources that the Fugue Conductor creates as part of executing fugue run on a composition. --include-fugue does not apply to the resources of the Fugue Conductor itself, which are included in the scan by default if the Conductor is installed in the scanned region.

Suppressing Transcriber output

To suppress Transcriber output, run Transcriber in quiet mode with the --quiet or -q option. The following example turns on quiet mode, scans the us-west-2 region, limits the results to VPCs, and sends the output to standard out:

fugue-transcriber -q --region us-west-2 -i aws-ec2-vpcs -

Since Transcriber is running in quiet mode, the only output is Ludwig:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2


############################################################
# VPC
#   Generated from AWS resource: vpc-922891f7
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-175bb772", region),
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
}