Transcriber Examples

Overview

This page contains examples using Fugue Transcriber. To read an overview of Transcriber or see details about usage, refer to the main Fugue Transcriber page here.

General Examples

Scanning a non-default region

Transcriber scans your default AWS region. If you’d like Transcriber to scan a different region, use the --region option. The example below scans the us-west-2 region and saves the output to composition.lw:

fugue-transcriber --region us-west-2 composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-west-2 region.
  Checking for Fugue runtime resources........ none found
  Checking for Fugue API server............... not found
  Scanning AutoScaling Groups................. found 0
  Scanning AutoScaling Launch Configurations.. found 0
  Scanning AutoScaling Scaling Policies....... found 0
  Scanning CloudFormation Stacks.............. found 0
  Scanning CloudWatch Alarms.................. found 0
  Scanning DynamoDB Tables.................... found 0
  Scanning EC2 Customer Gateways.............. found 0
  Scanning EC2 DHCP Options................... found 2
  Scanning EC2 Elastic IP Addresses........... found 0
  Scanning EC2 Instances...................... found 0
  Scanning EC2 Internet Gateways.............. found 1
  Scanning EC2 Network ACLs................... found 2
  Scanning EC2 Network Interfaces............. found 0
  Scanning EC2 Route Tables................... found 2
  Scanning EC2 Security Groups................ found 2
  Scanning EC2 Subnets........................ found 3
  Scanning EC2 Volumes........................ found 0
  Scanning EC2 VPC Endpoints.................. found 0
  Scanning EC2 VPC Peering Connections........ found 0
  Scanning EC2 VPCs........................... found 2
  Scanning EC2 VPN Connections................ found 0
  Scanning EC2 VPN Gateways................... found 0
  Scanning ELB Load Balancers................. found 0
  Scanning ELB Target Groups.................. found 1
  Scanning IAM Instance Profiles.............. found 5
  Scanning IAM Policies....................... found 11
  Scanning IAM Roles.......................... found 10
  Scanning Lambda Aliases..................... found 0
  Scanning Lambda Event Sources............... found 0
  Scanning Lambda Functions................... found 0
  Scanning RDS Instances...................... found 0
  Scanning RDS Subnet Groups.................. found 0
  Scanning S3 Buckets......................... found 1
  Scanning SNS Subscriptions.................. found 0
  Scanning SNS Topics......................... found 0
  Scanning SQS Queues......................... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.ELBv2 as ELBv2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3
import Fugue.Core.AWS.EC2 as Core.EC2


############################################################
# Region
#   From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-175bb772
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
  domainName: "us-west-2.compute.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
  resourceId: "dopt-175bb772",
}


############################################################
# DHCP Options
#   Generated from AWS resource: dopt-2d1a454b
############################################################
ec2-dhcpoptions-2: EC2.DhcpOptions.new {
  domainName: "us-west-2.compute.internal",
  domainNameServers: [
    "AmazonProvidedDNS",
  ],
  region: region,
  resourceId: "dopt-2d1a454b",
  tags: [
    AWS.tag("Application", "Hello World"),
  ],
}


############################################################
# InternetGateway
#   Generated from AWS resource: igw-dbd70cbe
############################################################
ec2-internetgateway-1: EC2.InternetGateway.new {
  region: region,
  resourceId: "igw-dbd70cbe",
  vpc: ec2-vpc-2,
}


[output trimmed for length]

(Output trimmed for length.)

Scanning an account using a non-default AWS credential profile

Transcriber scans the account associated with the default AWS credential profile. To scan an account using a non-default profile, use the --profile option. If the profile is associated with an IAM role granting cross-account access to a trusting account, Transcriber will scan the trusting account.

The example below specifies the ec2-access-only profile, scans only VPCs, and uses - to direct output to standard out:

fugue-transcriber --profile ec2-access-only -i aws-ec2-vpcs -

You’ll see output like this. Note how the second line says “Using AWS profile ‘ec2-access-only’”:

Transcriber starting...
Using AWS profile 'ec2-access-only'.
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ unauthorized
  Checking for Fugue API server............... not found
  Scanning EC2 VPCs........................... found 2

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-23fa035b
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-23fa035b",
  tags: [
    AWS.tag("fugue-transcriber:aws:cloudformation:logical-id", "FugueVpc"),
    AWS.tag("Name", "Fugue Conductor VPC"),
    AWS.tag("fugue-transcriber:aws:cloudformation:stack-name", "fugue"),
    AWS.tag("fugue-transcriber:aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/fugue/fac2b4c0-aabb-11e7-8213-50faeaabf0d1"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-2: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-76f2ab13",
}


------------------------------------------------------------
Transcriber complete.

Suppressing Transcriber output

To suppress Transcriber output, run Transcriber in quiet mode with the --quiet or -q option. The following example turns on quiet mode, scans the us-west-2 region, limits the results to VPCs, and sends the output to standard out:

fugue-transcriber -q --region us-west-2 -i aws-ec2-vpcs -

Since Transcriber is running in quiet mode, the only output is Ludwig:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2


############################################################
# VPC
#   Generated from AWS resource: vpc-922891f7
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-175bb772", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-922891f7",
}

RDS databases and password considerations

When transcribing an RDS database, because we cannot possibly know the password for the database, Transcriber will populate the field with DUMMYPASSWORD.

  • If you already know your database password and want to leave it as-is no action is required.
  • To update this password you can modify the password in the composition and issue a fugue update command.

EC2 and custom instance storage

In scenarios where the default instance storage for EC2 has been modified AWS does not provide visibility for these changes. As a result these changes cannot be transcribed and will need to be added manually.

For EC2 the following rules apply:

  • If you launch an instance type with no instance stores, Transcriber operates as expected.
  • If you launch an instance type with instance stores but you did not specify any details and launched using the default, Transcriber operates as expected.
  • If you launch an instance with custom mapping for instance stores, you will need to manually add the details from instanceStore field to any transcribed files.

For example:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.Pattern.Network as Network

network: Network.new {
  name: "EXAMPLE",
  cidr: "10.42.0.0/16",
  region: AWS.Us-east-1,
  publicSubnets: [
    (AWS.B, "10.42.12.0/24"),
    (AWS.C, "10.42.13.0/24")
  ],
  privateSubnets: [
    (AWS.B, "10.42.22.0/24"),
    (AWS.C, "10.42.23.0/24")
  ]
}

mySecurityGroup: EC2.SecurityGroup.new {
  description: "My Instance Security Group",
  ipPermissions: [
    # add rules here...
  ],
  ipPermissionsEgress: None,
  vpc: network.vpc,
}

myInstance: EC2.Instance.new {
  image: "ami-97785bed",
  instanceType: EC2.I3_large,
  securityGroups: [mySecurityGroup],
  subnet: List.index(network.publicSubnets, 0),

  # This is picked up by transcriber
  rootBlockDevice: EC2.RootBlockDevice.new {
    volumeSize: 500,
    volumeType: EC2.Gp2,
  },
  # This is picked up by transcriber
  volumes: [
    EC2.VolumeAttachment.new {
      volume: EC2.Volume.new{
        size:             16,
        availabilityZone: AWS.B,
        volumeType:       EC2.Gp2,
        region:           AWS.Us-east-1,
      },
      deviceName: "/dev/sdh",
    },
  ],
  # This is NOT picked up by transcriber and must be manually added.
  instanceStores: [
    EC2.InstanceStore.new {
      deviceName:  "/dev/sds",
      virtualName: "ephemeral0",
    },
  ],
}

Filtering Examples

Limiting results to resources with a specific tag key

If you want Transcriber to generate Ludwig only for resources tagged with a particular key, use the --tag-key or -k option. Transcriber uses this option to filter the results of the services scan. The example below searches for the tag key my-vpc and uses - to print to standard output:

fugue-transcriber --tag-key my-vpc -

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Checking for Fugue API server............... not found
  Scanning AutoScaling Groups................. found 0
  Scanning AutoScaling Launch Configurations.. found 0
  Scanning AutoScaling Scaling Policies....... found 0
  Scanning CloudFormation Stacks.............. found 0
  Scanning CloudWatch Alarms.................. found 0
  Scanning DynamoDB Tables.................... found 0
  Scanning EC2 Customer Gateways.............. found 0
  Scanning EC2 DHCP Options................... found 0
  Scanning EC2 Elastic IP Addresses........... found 0
  Scanning EC2 Instances...................... found 0
  Scanning EC2 Internet Gateways.............. found 0
  Scanning EC2 Network ACLs................... found 0
  Scanning EC2 Network Interfaces............. found 0
  Scanning EC2 Route Tables................... found 0
  Scanning EC2 Security Groups................ found 0
  Scanning EC2 Subnets........................ found 0
  Scanning EC2 Volumes........................ found 0
  Scanning EC2 VPC Endpoints.................. found 0
  Scanning EC2 VPC Peering Connections........ found 0
  Scanning EC2 VPCs........................... found 1
  Scanning EC2 VPN Connections................ found 0
  Scanning EC2 VPN Gateways................... found 0
  Scanning ELB Load Balancers................. found 0
  Scanning ELB Target Groups.................. found 0
  Scanning IAM Instance Profiles.............. found 0
  Scanning IAM Policies....................... found 0
  Scanning IAM Roles.......................... found 0
  Scanning Lambda Aliases..................... found 0
  Scanning Lambda Event Sources............... found 0
  Scanning Lambda Functions................... found 0
  Scanning RDS Instances...................... found 0
  Scanning RDS Subnet Groups.................. found 0
  Scanning S3 Buckets......................... found 0
  Scanning SNS Subscriptions.................. found 0
  Scanning SNS Topics......................... found 0
  Scanning SQS Queues......................... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-abb72cd3
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: False,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-abb72cd3",
  tags: [
    AWS.tag("Name", ""),
    AWS.tag("my-vpc", "My Awesome VPC"),
  ],
}


------------------------------------------------------------
Transcriber complete.

Limiting results to resources with a specific tag key-value pair

If you want Transcriber to generate Ludwig only for resources tagged with a particular key-value pair, use the --tag-pair or -p option. Transcriber uses this option to filter the results of the services scan. The example below searches for the tag key Name and tag value my-subnet, and uses - to print to standard output:

fugue-transcriber --tag-pair Name my-subnet -

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Checking for Fugue API server............... not found
  Scanning AutoScaling Groups................. found 0
  Scanning AutoScaling Launch Configurations.. found 0
  Scanning AutoScaling Scaling Policies....... found 0
  Scanning CloudFormation Stacks.............. found 0
  Scanning CloudWatch Alarms.................. found 0
  Scanning DynamoDB Tables.................... found 0
  Scanning EC2 Customer Gateways.............. found 0
  Scanning EC2 DHCP Options................... found 0
  Scanning EC2 Elastic IP Addresses........... found 0
  Scanning EC2 Instances...................... found 0
  Scanning EC2 Internet Gateways.............. found 0
  Scanning EC2 Network ACLs................... found 0
  Scanning EC2 Network Interfaces............. found 0
  Scanning EC2 Route Tables................... found 0
  Scanning EC2 Security Groups................ found 0
  Scanning EC2 Subnets........................ found 1
  Scanning EC2 Volumes........................ found 0
  Scanning EC2 VPC Endpoints.................. found 0
  Scanning EC2 VPC Peering Connections........ found 0
  Scanning EC2 VPCs........................... found 0
  Scanning EC2 VPN Connections................ found 0
  Scanning EC2 VPN Gateways................... found 0
  Scanning ELB Load Balancers................. found 0
  Scanning ELB Target Groups.................. found 0
  Scanning IAM Instance Profiles.............. found 0
  Scanning IAM Policies....................... found 0
  Scanning IAM Roles.......................... found 0
  Scanning Lambda Aliases..................... found 0
  Scanning Lambda Event Sources............... found 0
  Scanning Lambda Functions................... found 0
  Scanning RDS Instances...................... found 0
  Scanning RDS Subnet Groups.................. found 0
  Scanning S3 Buckets......................... found 0
  Scanning SNS Subscriptions.................. found 0
  Scanning SNS Topics......................... found 0
  Scanning SQS Queues......................... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# Subnet
#   Generated from AWS resource: subnet-26751742
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.B,
  cidrBlock: "10.0.0.0/24",
  defaultForAz: False,
  mapPublicIpOnLaunch: False,
  resourceId: "subnet-26751742",
  tags: [
    AWS.tag("Name", "my-subnet"),
  ],
  vpc: EC2.Vpc.external("vpc-abb72cd3", region),
}


------------------------------------------------------------
Transcriber complete.

Including only specific services in the results

To scan only certain AWS services, use the --include-service or -i option. It may be used multiple times in order to include multiple services. The following example limits Transcriber’s results to VPCs and subnets and saves the output to composition.lw:

fugue-transcriber -i aws-ec2-vpcs -i aws-ec2-subnets composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Checking for Fugue API server............... not found
  Scanning EC2 Subnets........................ found 7
  Scanning EC2 VPCs........................... found 3

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# Subnet
#   Generated from AWS resource: subnet-40951d19
############################################################
ec2-subnet-1: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "172.31.16.0/20",
  defaultForAz: True,
  mapPublicIpOnLaunch: True,
  resourceId: "subnet-40951d19",
  vpc: ec2-vpc-3,
}


############################################################
# Subnet
#   Generated from AWS resource: subnet-a3b7ddd4
############################################################
ec2-subnet-2: EC2.Subnet.new {
  availabilityZone: AWS.D,
  cidrBlock: "172.31.0.0/20",
  defaultForAz: True,
  mapPublicIpOnLaunch: True,
  resourceId: "subnet-a3b7ddd4",
  vpc: ec2-vpc-3,
}

[output trimmed for length]

(Output trimmed for length.)

For a list of supported services and how to refer to them in Transcriber, see Supported Services.

Excluding specific services from the results

To exclude certain AWS services from Transcriber’s scan, use the --exclude-service or -x option. It may be used multiple times to exclude multiple services. The following example excludes VPCs and subnets from the scan results and saves the output to composition.lw:

fugue-transcriber -x aws-ec2-vpcs -x aws-ec2-subnets composition.lw

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Checking for Fugue API server............... not found
  Scanning AutoScaling Groups................. found 0
  Scanning AutoScaling Launch Configurations.. found 0
  Scanning AutoScaling Scaling Policies....... found 0
  Scanning CloudFormation Stacks.............. found 0
  Scanning CloudWatch Alarms.................. found 8
  Scanning DynamoDB Tables.................... found 0
  Scanning EC2 Customer Gateways.............. found 0
  Scanning EC2 DHCP Options................... found 2
  Scanning EC2 Elastic IP Addresses........... found 0
  Scanning EC2 Instances...................... found 0
  Scanning EC2 Internet Gateways.............. found 1
  Scanning EC2 Network ACLs................... found 4
  Scanning EC2 Network Interfaces............. found 0
  Scanning EC2 Route Tables................... found 4
  Scanning EC2 Security Groups................ found 11
  Scanning EC2 Volumes........................ found 1
  Scanning EC2 VPC Endpoints.................. found 0
  Scanning EC2 VPC Peering Connections........ found 0
  Scanning EC2 VPN Connections................ found 0
  Scanning EC2 VPN Gateways................... found 0
  Scanning ELB Load Balancers................. found 0
  Scanning ELB Target Groups.................. found 0
  Scanning IAM Instance Profiles.............. found 5
  Scanning IAM Policies....................... found 11
  Scanning IAM Roles.......................... found 10
  Scanning Lambda Aliases..................... found 0
  Scanning Lambda Event Sources............... found 0
  Scanning Lambda Functions................... found 0
  Scanning RDS Instances...................... found 0
  Scanning RDS Subnet Groups.................. found 0
  Scanning S3 Buckets......................... found 2
  Scanning SNS Subscriptions.................. found 0
  Scanning SNS Topics......................... found 0
  Scanning SQS Queues......................... found 0

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.

And the generated composition, composition.lw, looks like this:

import Fugue.AWS.CloudWatch as CloudWatch
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3
import Fugue.AWS.SNS as SNS
import Fugue.Core.AWS.EC2 as Core.EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# Alarm
#   Generated from AWS resource: arn:aws:cloudwatch:us-east-1:123456789012:alarm:test-table-ReadCapacityUnitsLimit-BasicAlarm
############################################################
cloudwatch-alarm-1: CloudWatch.Alarm.new {
  actionsEnabled: True,
  alarmActions: [
    CloudWatch.Action.notifyTopic(SNS.Topic.external("arn:aws:sns:us-east-1:123456789012:dynamodb")),
  ],
  comparisonOperator: CloudWatch.GreaterThanOrEqualToThreshold,
  dimensions: {
    "TableName": "test-table",
  },
  evaluationPeriods: 5,
  metricName: "ConsumedReadCapacityUnits",
  name: "test-table-ReadCapacityUnitsLimit-BasicAlarm",
  namespace: "AWS/DynamoDB",
  period: 60,
  region: region,
  resourceId: "arn:aws:cloudwatch:us-east-1:123456789012:alarm:test-table-ReadCapacityUnitsLimit-BasicAlarm",
  statistic: CloudWatch.Sum,
  threshold: 48.0,
}

[output trimmed for length]

(Output trimmed for length.)

For a list of supported services and how to refer to them in Transcriber, see Supported Services.

Include resources managed by Fugue

When you fugue run a composition, Fugue creates a process and manages the resources declared in the composition. Transcriber uses the Fugue API server to filter out Fugue-managed resources, if the server is available. If the server is up and you’d like Transcriber to include Fugue-managed resources in its scan, use the --include-fugue-resources option. The following example presumes the API server is running on the default port (8080), includes Fugue-managed resources, and, for simplicity, only scans VPCs. The - directs output to standard out.

fugue-transcriber --include-fugue-resources --include-service aws-ec2-vpcs -

Fugue happens to be running a process that created a VPC named exampleVpc, so because we used the --include-fugue-resources option, that VPC is included in the results:

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Including Fugue-managed resources
  Scanning EC2 VPCs........................... found 2

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-fd62ff85
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec2f6895", region),
  enableDnsHostnames: False,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-fd62ff85",
  tags: [
    AWS.tag("Name", "exampleVpc"),
    AWS.tag("Application", "Fugue-managed resource"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-2: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-76f2ab13",
}


------------------------------------------------------------
Transcriber complete.

(Output trimmed for length.)

If the API server is still running and you run the same Transcriber command without --include-fugue-resources, you’ll see that exampleVpc is not included:

fugue-transcriber --include-service aws-ec2-vpcs -

Transcriber starting...
Scanning in us-east-1 region.
  Checking for Fugue runtime resources........ found
  Checking for Fugue API server............... found on port 8080
  Checking for Fugue managed resources........ found 7
  Scanning EC2 VPCs........................... found 1

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-76f2ab13",
}


------------------------------------------------------------
Transcriber complete.

Note: --include-fugue-resources only applies to resources tagged with a FID – resources that the Fugue Conductor creates as part of executing fugue run on a composition. --include-fugue-resources does not apply to the resources of the Fugue Conductor itself, which are included in the scan only if the --include-fugue-runtime option is set and the Conductor has been installed in the scanned region.

Including Fugue runtime resources

The Fugue Conductor has its own set of resources. If you’re scanning the region in which the Conductor has been installed, and you’d like to include the Fugue runtime resources, use the --include-fugue-runtime option. The following example scans the us-east-1 region, limits the results to VPCs, and sends the output to standard out:

fugue-transcriber --region us-east-1 -i aws-ec2-vpcs --include-fugue-runtime -

You’ll see output like this:

Transcriber starting...
Scanning in us-east-1 region.
  Including Fugue runtime resources
  Checking for Fugue API server............... not found
  Scanning EC2 VPCs........................... found 3

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2


############################################################
# Region
#   From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1


############################################################
# VPC
#   Generated from AWS resource: vpc-fd62ff85
############################################################
ec2-vpc-1: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec2f6895", region),
  enableDnsHostnames: False,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-fd62ff85",
  tags: [
    AWS.tag("Name", "exampleVpc"),
    AWS.tag("Application", "Fugue-managed resource"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-23fa035b
############################################################
ec2-vpc-2: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-23fa035b",
  tags: [
    AWS.tag("fugue-transcriber:aws:cloudformation:logical-id", "FugueVpc"),
    AWS.tag("Name", "Fugue Conductor VPC"),
    AWS.tag("fugue-transcriber:aws:cloudformation:stack-name", "fugue"),
    AWS.tag("fugue-transcriber:aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:141874191075:stack/fugue/fac2b4c0-aabb-11e7-8213-50faeaabf0d1"),
  ],
}


############################################################
# VPC
#   Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-3: EC2.Vpc.new {
  cidrBlock: "172.31.0.0/16",
  dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
  enableDnsHostnames: True,
  enableDnsSupport: True,
  instanceTenancy: EC2.DefaultTenancy,
  region: region,
  resourceId: "vpc-76f2ab13",
}


------------------------------------------------------------
Transcriber complete.

Note how ec2-vpc-2 has a tag value of Fugue Conductor VPC.

Advanced Filtering Examples

In addition to the filtering capabilities demonstrated above, Transcriber offers advanced filtering with the --filter-file option. In this experimental feature, Transcriber scans services based on a user-provided YAML file containing an array of filters. This enables users to be more concise and specific about which resources are transcribed.

Warning

IMPORTANT NOTE: Transcriber’s advanced filter functionality is experimental and should be approached with caution. If you are unsure about deploying this type of filtering reach out to support@fugue.co.

For example, you can filter on resource-specific properties, like using VpcId to return all of the subnets in a given VPC, or using UserName to exclude your admin IAM role from transcription. You can also use regular expressions, operators, and JMESPath to fine-tune a filter.

Advanced filtering doesn’t have to be complex, though. Any standard filter can be written as an advanced filter. Another benefit of advanced filtering is that you can include/exclude multiple services in a file rather than specifying each one at the command line. Note: We don’t recommend using both standard and advanced filtering options in the same command.

To identify the specific properties you may want to filter on, you can examine the results of the AWS CLI describe calls (e.g., aws ec2 describe-vpcs, aws iam list-users). If a field is listed in the output of a describe call for a supported service, you can filter on it. Some useful properties are RoleId, Arn, and fields with the word Name in them, among others.

Writing Transcriber Filters

Each filter in the file should start on a new line. Transcriber supports two different filter formats:

- <filter-type>: <transcriber-service-name>:<jmespath-expression><operator><pattern>

And:

- <filter-type><transcriber-service-name>:*
  • <filter-type> is either include or exclude. include indicates that Transcriber should include the specified service or resource when it queries AWS, and exclude means that Transcriber excludes it. If you include services, all others are implicitly excluded. If you exclude services, all others are implicitly included.
  • <transcriber-service-name> specifies the name of the AWS service as it appears in Transcriber’s service list. You can view this list by executing fugue-transcriber --list-services or fugue-transcriber -l. Use * as a wildcard (as seen in the second example above).
  • <jmespath-expression> is a JMESPath expression indicating a specific resource property. Since AWS API responses are formatted as JSON, Transcriber allows you to use JMESPath expressions to refine the results. For example, Tags[].Value uses the flatten operator to return the Value for each item in the Tags list. Note: JMESPath support is not yet complete. If you encounter issues, reach out to support@fugue.co.
  • <operator> indicates the type of operation that should be performed. There are several supported operators:
    • eq or == (equal to)
    • ge or >= (greater than or equal to)
    • gt or > (greater than)
    • le or <= (less than or equal to)
    • lt or < (less than)
    • ne or != (not equal to)
    • re or ~= (regular expression, or regex)
  • <pattern> only applies if you’re using the regex operator. It’s a scalar value or regular expression.

Using the Advanced Filtering Feature

With the above guidelines in mind, let’s try an example. You’ll need to download six files from our Github repo:

We’ll fugue run the composition to instantiate infrastructure in the us-west-2 region, then use Transcriber to scan the region using the YAML file filters. The four JSON files provide IAM policies used in the composition.

TranscriberFilterComposition.lw creates 1 DHCP options set, 1 VPC, 2 security groups with 4 rules, 1 Elastic IP address, 2 DynamoDB tables, 2 SQS queues with policy, 2 SNS topics, 1 SNS subscription, 1 IAM policy, 1 IAM managed policy, and 1 IAM role.

Before we get started, change the AWS account number on line 12 of TranscriberFilterComposition.lw. Replace the dummy string with your own AWS account ID.

Next, run the composition and give it the alias filterTest:

fugue run TranscriberFilterComposition.lw --alias filterTest

When fugue status shows the process has successfully been created, execute the following Transcriber command:

fugue-transcriber --region us-west-2 --filter-file TranscriberFilter.yaml -

This command instructs Transcriber to scan your AWS account in the us-west-2 region using the filter file TranscriberFilter.yaml, then output the results to screen. Note: You can also use -f to set the filter file.

You’ll see output like this:

Transcriber starting...
Scanning in us-west-2 region.
  Checking for Fugue runtime resources........ not found
  Checking for Fugue API server............... not found
  Scanning DynamoDB Tables.................... found 2
  Scanning EC2 DHCP Options................... found 1
  Scanning EC2 Elastic IP Addresses........... found 1
  Scanning EC2 Security Groups................ found 2
  Scanning EC2 VPCs........................... found 1
  Scanning IAM Managed Policies............... found 1
  Scanning IAM Roles.......................... found 1
  Scanning SNS Subscriptions.................. found 1
  Scanning SNS Topics......................... found 2
  Scanning SQS Queues......................... found 2

------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------

[output trimmed for length]

Output trimmed for length.

The output will follow the same format as any other Transcriber run. Skim the output, and you’ll see that the filters in TranscriberFilter.yaml ensure Transcriber returns only the infrastructure we created with TranscriberFilterComposition.lw.

Let’s look at the filters more closely. We’ll start with line 8:

- include: aws-ec2-elastic-ip-addresses:*

This is a very simple filter that returns all of the Elastic IP addresses Transcriber scans in a region. The filter type is include, the service to scan is aws-ec2-elastic-ip-addresses, and the operator is the wild card *, so all Elastic IP addresses are listed.

Here’s a more advanced filter on line 2:

- include: aws-ec2-dhcp-options:Tags[].Key == transcriber-test

This filter returns all of the DHCP options sets with a tag key that matches transcriber-test. The filter type is include, the service to scan is aws-ec2-dhcp-options, the JMESPath expression is Tags[].Key (which flattens the Tags list and returns each Key in the list), the operator is ==, and the pattern is transcriber-test.

Line 16 demonstrates a filter that uses regex:

- include: aws-sns-topics:TopicArn ~= .*:topic[12]

The filter returns only the SNS topics with :topic1 or :topic2 in the ARN. The filter type is include, the service to scan is aws-sns-topics, the JMESPath expression is TopicArn, the operator is ~=, and the regex expression is .*:topic[12].

That covers the basics of writing filters. Because the filter file is written as YAML, you can comment out lines by prepending a # symbol. Feel free to comment and uncomment lines to try out different combinations of filters.

When you’re all done experimenting, just execute fugue kill filterTest to terminate the process infrastructure.

Tips for Advanced Filtering

  • There’s a quick way to find all resources across all services that match a certain tag name. The convenience value is tag:name, and it’s short for *:Tags[?Key=='Name'].Value || TagSet[?Key=='Name'].Value. Here’s one way to use it:
- include: tag:name == vpc
  • Dates are in ISO 8601 format.
  • Your filter file can contain both include and exclude filters. Resources accepted by include filters will be passed through exclude filters. If no include filters are specified, all resources will be passed through exclude filters.
  • Start all filters with a hyphen at the beginning of the line.
  • You can filter on reference to a particular resource. This is useful for filtering on a unique property. Here’s an example that will list all the instances inside vpc-76f2ab13:
- include: aws-ec2-instances:VpcId == vpc-76f2ab13
  • Filters can be combined. The following example searches for all security groups that have a tag key matching transcriber-test and have no ingress rules (meaning the security groups have an IP protocol of -1):
- include:
  - aws-ec2-security-groups:Tags[].Key == transcriber-test
  - aws-ec2-security-groups:IpPermissionsEgress[].IpProtocol == -1

If you transcribe with this filter while the filterTest process is running, Transcriber will return just sg1, which is the security group with the description “Empty SG.”

Caveats for Advanced Filtering

As we mentioned earlier, advanced filtering is an experimental feature. The list of caveats below is not intended to be exhaustive. If you encounter unexpected behavior, reach out to support@fugue.co with any questions.

You can filter on any AWS field returned in the describe or list call for a Transcriber service. However, some services’ describe and list calls don’t return a lot of info, so other approaches are required:

SQS: The aws sqs list-queues call returns QueueUrls URLs and nothing else. However, once you have a queue URL, you can execute a second call to return a wider list of properties you can filter on:

aws sqs get-queue-attributes --queue-url QUEUE_URL --attribute-names All

SNS: The aws sns list-topics call returns TopicArn ARNs and nothing else. At this time, that’s the only topic property you can filter on. You can use regex to return just the topics that match a given pattern, as we showed in the previous section. You can also use the ARN to return other resources that refer to it. This filter uses regex to return all the SNS subscriptions that refer to SNS topics ending in :topic1:

- include: aws-sns-subscriptions:TopicArn ~= .*:topic1

A Word of Caution

Transcriber is a read-only service, and scanning your account won’t affect the infrastructure in it. That said, the composition Transcriber creates through advanced filtering should not be used to launch a process or import your infrastructure. Advanced filtering is currently experimental. We cannot guarantee that a composition generated with advanced filtering will have no side effects when it’s used with fugue run or fugue run --import in production.