Transcriber Examples¶
Overview¶
This page contains examples using Fugue Transcriber. To read an overview of Transcriber or see details about usage, refer to the main Fugue Transcriber page here.
Table of Contents¶
- Scanning a non-default region
- Scanning an account using a non-default AWS credential profile
- Suppressing Transcriber output
- RDS databases and password considerations
- EC2 and custom instance storage
General Examples¶
Scanning a non-default region¶
Transcriber scans your default AWS region. If you’d like Transcriber to
scan a different region, use the --region
option. The example below
scans the us-west-2
region and saves the output to
composition.lw
:
fugue-transcriber --region us-west-2 composition.lw
You’ll see output like this:
Transcriber starting...
Scanning in us-west-2 region.
Checking for Fugue runtime resources........ none found
Checking for Fugue API server............... not found
Scanning AutoScaling Groups................. found 0
Scanning AutoScaling Launch Configurations.. found 0
Scanning AutoScaling Scaling Policies....... found 0
Scanning CloudFormation Stacks.............. found 0
Scanning CloudWatch Alarms.................. found 0
Scanning DynamoDB Tables.................... found 0
Scanning EC2 Customer Gateways.............. found 0
Scanning EC2 DHCP Options................... found 2
Scanning EC2 Elastic IP Addresses........... found 0
Scanning EC2 Instances...................... found 0
Scanning EC2 Internet Gateways.............. found 1
Scanning EC2 Network ACLs................... found 2
Scanning EC2 Network Interfaces............. found 0
Scanning EC2 Route Tables................... found 2
Scanning EC2 Security Groups................ found 2
Scanning EC2 Subnets........................ found 3
Scanning EC2 Volumes........................ found 0
Scanning EC2 VPC Endpoints.................. found 0
Scanning EC2 VPC Peering Connections........ found 0
Scanning EC2 VPCs........................... found 2
Scanning EC2 VPN Connections................ found 0
Scanning EC2 VPN Gateways................... found 0
Scanning ELB Load Balancers................. found 0
Scanning ELB Target Groups.................. found 1
Scanning IAM Instance Profiles.............. found 5
Scanning IAM Policies....................... found 11
Scanning IAM Roles.......................... found 10
Scanning Lambda Aliases..................... found 0
Scanning Lambda Event Sources............... found 0
Scanning Lambda Functions................... found 0
Scanning RDS Instances...................... found 0
Scanning RDS Subnet Groups.................. found 0
Scanning S3 Buckets......................... found 1
Scanning SNS Subscriptions.................. found 0
Scanning SNS Topics......................... found 0
Scanning SQS Queues......................... found 0
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.
And the generated composition, composition.lw
, looks like this:
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.ELBv2 as ELBv2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3
import Fugue.Core.AWS.EC2 as Core.EC2
############################################################
# Region
# From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2
############################################################
# DHCP Options
# Generated from AWS resource: dopt-175bb772
############################################################
ec2-dhcpoptions-1: EC2.DhcpOptions.new {
domainName: "us-west-2.compute.internal",
domainNameServers: [
"AmazonProvidedDNS",
],
region: region,
resourceId: "dopt-175bb772",
}
############################################################
# DHCP Options
# Generated from AWS resource: dopt-2d1a454b
############################################################
ec2-dhcpoptions-2: EC2.DhcpOptions.new {
domainName: "us-west-2.compute.internal",
domainNameServers: [
"AmazonProvidedDNS",
],
region: region,
resourceId: "dopt-2d1a454b",
tags: [
AWS.tag("Application", "Hello World"),
],
}
############################################################
# InternetGateway
# Generated from AWS resource: igw-dbd70cbe
############################################################
ec2-internetgateway-1: EC2.InternetGateway.new {
region: region,
resourceId: "igw-dbd70cbe",
vpc: ec2-vpc-2,
}
[output trimmed for length]
(Output trimmed for length.)
Scanning an account using a non-default AWS credential profile¶
Transcriber scans the account associated with the default AWS credential
profile.
To scan an account using a non-default profile, use the --profile
option. If the profile is associated with an IAM role granting
cross-account access to a trusting
account,
Transcriber will scan the trusting account.
The example below specifies the ec2-access-only
profile, scans only
VPCs, and uses -
to direct output to standard out:
fugue-transcriber --profile ec2-access-only -i aws-ec2-vpcs -
You’ll see output like this. Note how the second line says “Using AWS profile ‘ec2-access-only’”:
Transcriber starting...
Using AWS profile 'ec2-access-only'.
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ unauthorized
Checking for Fugue API server............... not found
Scanning EC2 VPCs........................... found 2
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# VPC
# Generated from AWS resource: vpc-23fa035b
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "10.0.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-23fa035b",
tags: [
AWS.tag("fugue-transcriber:aws:cloudformation:logical-id", "FugueVpc"),
AWS.tag("Name", "Fugue Conductor VPC"),
AWS.tag("fugue-transcriber:aws:cloudformation:stack-name", "fugue"),
AWS.tag("fugue-transcriber:aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/fugue/fac2b4c0-aabb-11e7-8213-50faeaabf0d1"),
],
}
############################################################
# VPC
# Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-2: EC2.Vpc.new {
cidrBlock: "172.31.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-76f2ab13",
}
------------------------------------------------------------
Transcriber complete.
Suppressing Transcriber output¶
To suppress Transcriber output, run Transcriber in quiet mode with the
--quiet
or -q
option. The following example turns on quiet mode,
scans the us-west-2
region, limits the results to VPCs, and sends
the output to standard out:
fugue-transcriber -q --region us-west-2 -i aws-ec2-vpcs -
Since Transcriber is running in quiet mode, the only output is Ludwig:
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-west-2
############################################################
region: AWS.Us-west-2
############################################################
# VPC
# Generated from AWS resource: vpc-922891f7
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "172.31.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-175bb772", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-922891f7",
}
RDS databases and password considerations¶
When transcribing an RDS database, because we cannot possibly know the
password for the database, Transcriber will populate the field with
DUMMYPASSWORD
.
- If you already know your database password and want to leave it as-is no action is required.
- To update this password you can modify the password in the
composition and issue a
fugue update
command.
EC2 and custom instance storage¶
In scenarios where the default instance storage for EC2 has been modified AWS does not provide visibility for these changes. As a result these changes cannot be transcribed and will need to be added manually.
For EC2 the following rules apply:
- If you launch an instance type with no instance stores, Transcriber operates as expected.
- If you launch an instance type with instance stores but you did not specify any details and launched using the default, Transcriber operates as expected.
- If you launch an instance with custom mapping for instance stores,
you will need to manually add the details from
instanceStore
field to any transcribed files.
For example:
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.Pattern.Network as Network
network: Network.new {
name: "EXAMPLE",
cidr: "10.42.0.0/16",
region: AWS.Us-east-1,
publicSubnets: [
(AWS.B, "10.42.12.0/24"),
(AWS.C, "10.42.13.0/24")
],
privateSubnets: [
(AWS.B, "10.42.22.0/24"),
(AWS.C, "10.42.23.0/24")
]
}
mySecurityGroup: EC2.SecurityGroup.new {
description: "My Instance Security Group",
ipPermissions: [
# add rules here...
],
ipPermissionsEgress: None,
vpc: network.vpc,
}
myInstance: EC2.Instance.new {
image: "ami-97785bed",
instanceType: EC2.I3_large,
securityGroups: [mySecurityGroup],
subnet: List.index(network.publicSubnets, 0),
# This is picked up by transcriber
rootBlockDevice: EC2.RootBlockDevice.new {
volumeSize: 500,
volumeType: EC2.Gp2,
},
# This is picked up by transcriber
volumes: [
EC2.VolumeAttachment.new {
volume: EC2.Volume.new{
size: 16,
availabilityZone: AWS.B,
volumeType: EC2.Gp2,
region: AWS.Us-east-1,
},
deviceName: "/dev/sdh",
},
],
# This is NOT picked up by transcriber and must be manually added.
instanceStores: [
EC2.InstanceStore.new {
deviceName: "/dev/sds",
virtualName: "ephemeral0",
},
],
}
Filtering Examples¶
Limiting results to resources with a specific tag key¶
If you want Transcriber to generate Ludwig only for resources tagged
with a particular key, use the --tag-key
or -k
option.
Transcriber uses this option to filter the results of the services scan.
The example below searches for the tag key my-vpc
and uses -
to
print to standard output:
fugue-transcriber --tag-key my-vpc -
You’ll see output like this:
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Checking for Fugue API server............... not found
Scanning AutoScaling Groups................. found 0
Scanning AutoScaling Launch Configurations.. found 0
Scanning AutoScaling Scaling Policies....... found 0
Scanning CloudFormation Stacks.............. found 0
Scanning CloudWatch Alarms.................. found 0
Scanning DynamoDB Tables.................... found 0
Scanning EC2 Customer Gateways.............. found 0
Scanning EC2 DHCP Options................... found 0
Scanning EC2 Elastic IP Addresses........... found 0
Scanning EC2 Instances...................... found 0
Scanning EC2 Internet Gateways.............. found 0
Scanning EC2 Network ACLs................... found 0
Scanning EC2 Network Interfaces............. found 0
Scanning EC2 Route Tables................... found 0
Scanning EC2 Security Groups................ found 0
Scanning EC2 Subnets........................ found 0
Scanning EC2 Volumes........................ found 0
Scanning EC2 VPC Endpoints.................. found 0
Scanning EC2 VPC Peering Connections........ found 0
Scanning EC2 VPCs........................... found 1
Scanning EC2 VPN Connections................ found 0
Scanning EC2 VPN Gateways................... found 0
Scanning ELB Load Balancers................. found 0
Scanning ELB Target Groups.................. found 0
Scanning IAM Instance Profiles.............. found 0
Scanning IAM Policies....................... found 0
Scanning IAM Roles.......................... found 0
Scanning Lambda Aliases..................... found 0
Scanning Lambda Event Sources............... found 0
Scanning Lambda Functions................... found 0
Scanning RDS Instances...................... found 0
Scanning RDS Subnet Groups.................. found 0
Scanning S3 Buckets......................... found 0
Scanning SNS Subscriptions.................. found 0
Scanning SNS Topics......................... found 0
Scanning SQS Queues......................... found 0
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# VPC
# Generated from AWS resource: vpc-abb72cd3
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "10.0.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: False,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-abb72cd3",
tags: [
AWS.tag("Name", ""),
AWS.tag("my-vpc", "My Awesome VPC"),
],
}
------------------------------------------------------------
Transcriber complete.
Limiting results to resources with a specific tag key-value pair¶
If you want Transcriber to generate Ludwig only for resources tagged
with a particular key-value pair, use the --tag-pair
or -p
option. Transcriber uses this option to filter the results of the
services scan. The example below searches for the tag key Name
and
tag value my-subnet
, and uses -
to print to standard output:
fugue-transcriber --tag-pair Name my-subnet -
You’ll see output like this:
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Checking for Fugue API server............... not found
Scanning AutoScaling Groups................. found 0
Scanning AutoScaling Launch Configurations.. found 0
Scanning AutoScaling Scaling Policies....... found 0
Scanning CloudFormation Stacks.............. found 0
Scanning CloudWatch Alarms.................. found 0
Scanning DynamoDB Tables.................... found 0
Scanning EC2 Customer Gateways.............. found 0
Scanning EC2 DHCP Options................... found 0
Scanning EC2 Elastic IP Addresses........... found 0
Scanning EC2 Instances...................... found 0
Scanning EC2 Internet Gateways.............. found 0
Scanning EC2 Network ACLs................... found 0
Scanning EC2 Network Interfaces............. found 0
Scanning EC2 Route Tables................... found 0
Scanning EC2 Security Groups................ found 0
Scanning EC2 Subnets........................ found 1
Scanning EC2 Volumes........................ found 0
Scanning EC2 VPC Endpoints.................. found 0
Scanning EC2 VPC Peering Connections........ found 0
Scanning EC2 VPCs........................... found 0
Scanning EC2 VPN Connections................ found 0
Scanning EC2 VPN Gateways................... found 0
Scanning ELB Load Balancers................. found 0
Scanning ELB Target Groups.................. found 0
Scanning IAM Instance Profiles.............. found 0
Scanning IAM Policies....................... found 0
Scanning IAM Roles.......................... found 0
Scanning Lambda Aliases..................... found 0
Scanning Lambda Event Sources............... found 0
Scanning Lambda Functions................... found 0
Scanning RDS Instances...................... found 0
Scanning RDS Subnet Groups.................. found 0
Scanning S3 Buckets......................... found 0
Scanning SNS Subscriptions.................. found 0
Scanning SNS Topics......................... found 0
Scanning SQS Queues......................... found 0
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# Subnet
# Generated from AWS resource: subnet-26751742
############################################################
ec2-subnet-1: EC2.Subnet.new {
availabilityZone: AWS.B,
cidrBlock: "10.0.0.0/24",
defaultForAz: False,
mapPublicIpOnLaunch: False,
resourceId: "subnet-26751742",
tags: [
AWS.tag("Name", "my-subnet"),
],
vpc: EC2.Vpc.external("vpc-abb72cd3", region),
}
------------------------------------------------------------
Transcriber complete.
Including only specific services in the results¶
To scan only certain AWS services, use the --include-service
or
-i
option. It may be used multiple times in order to include
multiple services. The following example limits Transcriber’s results to
VPCs and subnets and saves the output to composition.lw
:
fugue-transcriber -i aws-ec2-vpcs -i aws-ec2-subnets composition.lw
You’ll see output like this:
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Checking for Fugue API server............... not found
Scanning EC2 Subnets........................ found 7
Scanning EC2 VPCs........................... found 3
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.
And the generated composition, composition.lw
, looks like this:
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# Subnet
# Generated from AWS resource: subnet-40951d19
############################################################
ec2-subnet-1: EC2.Subnet.new {
availabilityZone: AWS.A,
cidrBlock: "172.31.16.0/20",
defaultForAz: True,
mapPublicIpOnLaunch: True,
resourceId: "subnet-40951d19",
vpc: ec2-vpc-3,
}
############################################################
# Subnet
# Generated from AWS resource: subnet-a3b7ddd4
############################################################
ec2-subnet-2: EC2.Subnet.new {
availabilityZone: AWS.D,
cidrBlock: "172.31.0.0/20",
defaultForAz: True,
mapPublicIpOnLaunch: True,
resourceId: "subnet-a3b7ddd4",
vpc: ec2-vpc-3,
}
[output trimmed for length]
(Output trimmed for length.)
For a list of supported services and how to refer to them in Transcriber, see Supported Services.
Excluding specific services from the results¶
To exclude certain AWS services from Transcriber’s scan, use the
--exclude-service
or -x
option. It may be used multiple times to
exclude multiple services. The following example excludes VPCs and
subnets from the scan results and saves the output to
composition.lw
:
fugue-transcriber -x aws-ec2-vpcs -x aws-ec2-subnets composition.lw
You’ll see output like this:
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Checking for Fugue API server............... not found
Scanning AutoScaling Groups................. found 0
Scanning AutoScaling Launch Configurations.. found 0
Scanning AutoScaling Scaling Policies....... found 0
Scanning CloudFormation Stacks.............. found 0
Scanning CloudWatch Alarms.................. found 8
Scanning DynamoDB Tables.................... found 0
Scanning EC2 Customer Gateways.............. found 0
Scanning EC2 DHCP Options................... found 2
Scanning EC2 Elastic IP Addresses........... found 0
Scanning EC2 Instances...................... found 0
Scanning EC2 Internet Gateways.............. found 1
Scanning EC2 Network ACLs................... found 4
Scanning EC2 Network Interfaces............. found 0
Scanning EC2 Route Tables................... found 4
Scanning EC2 Security Groups................ found 11
Scanning EC2 Volumes........................ found 1
Scanning EC2 VPC Endpoints.................. found 0
Scanning EC2 VPC Peering Connections........ found 0
Scanning EC2 VPN Connections................ found 0
Scanning EC2 VPN Gateways................... found 0
Scanning ELB Load Balancers................. found 0
Scanning ELB Target Groups.................. found 0
Scanning IAM Instance Profiles.............. found 5
Scanning IAM Policies....................... found 11
Scanning IAM Roles.......................... found 10
Scanning Lambda Aliases..................... found 0
Scanning Lambda Event Sources............... found 0
Scanning Lambda Functions................... found 0
Scanning RDS Instances...................... found 0
Scanning RDS Subnet Groups.................. found 0
Scanning S3 Buckets......................... found 2
Scanning SNS Subscriptions.................. found 0
Scanning SNS Topics......................... found 0
Scanning SQS Queues......................... found 0
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
Ludwig written to file: composition.lw
------------------------------------------------------------
Transcriber complete.
And the generated composition, composition.lw
, looks like this:
import Fugue.AWS.CloudWatch as CloudWatch
import Fugue.AWS.EC2 as EC2
import Fugue.AWS.IAM as IAM
import Fugue.AWS.S3 as S3
import Fugue.AWS.SNS as SNS
import Fugue.Core.AWS.EC2 as Core.EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# Alarm
# Generated from AWS resource: arn:aws:cloudwatch:us-east-1:123456789012:alarm:test-table-ReadCapacityUnitsLimit-BasicAlarm
############################################################
cloudwatch-alarm-1: CloudWatch.Alarm.new {
actionsEnabled: True,
alarmActions: [
CloudWatch.Action.notifyTopic(SNS.Topic.external("arn:aws:sns:us-east-1:123456789012:dynamodb")),
],
comparisonOperator: CloudWatch.GreaterThanOrEqualToThreshold,
dimensions: {
"TableName": "test-table",
},
evaluationPeriods: 5,
metricName: "ConsumedReadCapacityUnits",
name: "test-table-ReadCapacityUnitsLimit-BasicAlarm",
namespace: "AWS/DynamoDB",
period: 60,
region: region,
resourceId: "arn:aws:cloudwatch:us-east-1:123456789012:alarm:test-table-ReadCapacityUnitsLimit-BasicAlarm",
statistic: CloudWatch.Sum,
threshold: 48.0,
}
[output trimmed for length]
(Output trimmed for length.)
For a list of supported services and how to refer to them in Transcriber, see Supported Services.
Include resources managed by Fugue¶
When you fugue run
a composition, Fugue creates a
process and manages the resources declared in the composition.
Transcriber uses the Fugue API server to filter out
Fugue-managed resources, if the server is available. If the server is up
and you’d like Transcriber to include Fugue-managed resources in its
scan, use the --include-fugue-resources
option. The following
example presumes the API server is running on the default port (8080),
includes Fugue-managed resources, and, for simplicity, only scans VPCs.
The -
directs output to standard out.
fugue-transcriber --include-fugue-resources --include-service aws-ec2-vpcs -
Fugue happens to be running a process that created a VPC named
exampleVpc
, so because we used the --include-fugue-resources
option, that VPC is included in the results:
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Including Fugue-managed resources
Scanning EC2 VPCs........................... found 2
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# VPC
# Generated from AWS resource: vpc-fd62ff85
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "10.0.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec2f6895", region),
enableDnsHostnames: False,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-fd62ff85",
tags: [
AWS.tag("Name", "exampleVpc"),
AWS.tag("Application", "Fugue-managed resource"),
],
}
############################################################
# VPC
# Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-2: EC2.Vpc.new {
cidrBlock: "172.31.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-76f2ab13",
}
------------------------------------------------------------
Transcriber complete.
(Output trimmed for length.)
If the API server is still running and you run the same Transcriber
command without --include-fugue-resources
, you’ll see that
exampleVpc
is not included:
fugue-transcriber --include-service aws-ec2-vpcs -
Transcriber starting...
Scanning in us-east-1 region.
Checking for Fugue runtime resources........ found
Checking for Fugue API server............... found on port 8080
Checking for Fugue managed resources........ found 7
Scanning EC2 VPCs........................... found 1
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# VPC
# Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "172.31.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-76f2ab13",
}
------------------------------------------------------------
Transcriber complete.
Note: --include-fugue-resources
only applies to resources tagged
with a FID – resources that the Fugue Conductor creates as part of
executing fugue run
on a composition. --include-fugue-resources
does not apply to the resources of the Fugue
Conductor
itself, which are included in the scan only if the
--include-fugue-runtime
option is set and the Conductor has been
installed in the scanned region.
Including Fugue runtime resources¶
The Fugue Conductor has its own set of
resources.
If you’re scanning the region in which the Conductor has been installed,
and you’d like to include the Fugue runtime resources, use the
--include-fugue-runtime
option. The following example scans the
us-east-1
region, limits the results to VPCs, and sends the output
to standard out:
fugue-transcriber --region us-east-1 -i aws-ec2-vpcs --include-fugue-runtime -
You’ll see output like this:
Transcriber starting...
Scanning in us-east-1 region.
Including Fugue runtime resources
Checking for Fugue API server............... not found
Scanning EC2 VPCs........................... found 3
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
composition
import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2
############################################################
# Region
# From local AWS setting: us-east-1
############################################################
region: AWS.Us-east-1
############################################################
# VPC
# Generated from AWS resource: vpc-fd62ff85
############################################################
ec2-vpc-1: EC2.Vpc.new {
cidrBlock: "10.0.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec2f6895", region),
enableDnsHostnames: False,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-fd62ff85",
tags: [
AWS.tag("Name", "exampleVpc"),
AWS.tag("Application", "Fugue-managed resource"),
],
}
############################################################
# VPC
# Generated from AWS resource: vpc-23fa035b
############################################################
ec2-vpc-2: EC2.Vpc.new {
cidrBlock: "10.0.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-23fa035b",
tags: [
AWS.tag("fugue-transcriber:aws:cloudformation:logical-id", "FugueVpc"),
AWS.tag("Name", "Fugue Conductor VPC"),
AWS.tag("fugue-transcriber:aws:cloudformation:stack-name", "fugue"),
AWS.tag("fugue-transcriber:aws:cloudformation:stack-id", "arn:aws:cloudformation:us-east-1:141874191075:stack/fugue/fac2b4c0-aabb-11e7-8213-50faeaabf0d1"),
],
}
############################################################
# VPC
# Generated from AWS resource: vpc-76f2ab13
############################################################
ec2-vpc-3: EC2.Vpc.new {
cidrBlock: "172.31.0.0/16",
dhcpOptions: EC2.DhcpOptions.external("dopt-ec03ef89", region),
enableDnsHostnames: True,
enableDnsSupport: True,
instanceTenancy: EC2.DefaultTenancy,
region: region,
resourceId: "vpc-76f2ab13",
}
------------------------------------------------------------
Transcriber complete.
Note how ec2-vpc-2
has a tag value of Fugue Conductor VPC
.
Advanced Filtering Examples¶
In addition to the filtering capabilities demonstrated
above, Transcriber
offers advanced filtering with the --filter-file
option. In this
experimental feature, Transcriber scans services based on a
user-provided YAML file containing an array of filters. This enables
users to be more concise and specific about which resources are
transcribed.
Warning
IMPORTANT NOTE: Transcriber’s advanced filter functionality is experimental and should be approached with caution. If you are unsure about deploying this type of filtering reach out to support@fugue.co.
For example, you can filter on resource-specific properties, like using
VpcId
to return all of the subnets in a given VPC, or using
UserName
to exclude your admin IAM role from transcription. You can
also use regular expressions, operators, and JMESPath to fine-tune a
filter.
Advanced filtering doesn’t have to be complex, though. Any standard filter can be written as an advanced filter. Another benefit of advanced filtering is that you can include/exclude multiple services in a file rather than specifying each one at the command line. Note: We don’t recommend using both standard and advanced filtering options in the same command.
To identify the specific properties you may want to filter on, you can
examine the results of the AWS CLI describe calls (e.g.,
aws ec2 describe-vpcs
, aws iam list-users
). If a field is listed
in the output of a describe call for a supported service, you can filter
on it. Some useful properties are RoleId
, Arn
, and fields with
the word Name
in them, among others.
Writing Transcriber Filters¶
Each filter in the file should start on a new line. Transcriber supports two different filter formats:
- <filter-type>: <transcriber-service-name>:<jmespath-expression><operator><pattern>
And:
- <filter-type><transcriber-service-name>:*
<filter-type>
is eitherinclude
orexclude
.include
indicates that Transcriber should include the specified service or resource when it queries AWS, andexclude
means that Transcriber excludes it. If youinclude
services, all others are implicitly excluded. If youexclude
services, all others are implicitly included.<transcriber-service-name>
specifies the name of the AWS service as it appears in Transcriber’s service list. You can view this list by executingfugue-transcriber --list-services
orfugue-transcriber -l
. Use*
as a wildcard (as seen in the second example above).<jmespath-expression>
is a JMESPath expression indicating a specific resource property. Since AWS API responses are formatted as JSON, Transcriber allows you to use JMESPath expressions to refine the results. For example,Tags[].Value
uses the flatten operator to return theValue
for each item in theTags
list. Note: JMESPath support is not yet complete. If you encounter issues, reach out to support@fugue.co.<operator>
indicates the type of operation that should be performed. There are several supported operators:eq
or==
(equal to)ge
or>=
(greater than or equal to)gt
or>
(greater than)le
or<=
(less than or equal to)lt
or<
(less than)ne
or!=
(not equal to)re
or~=
(regular expression, or regex)
<pattern>
only applies if you’re using the regex operator. It’s a scalar value or regular expression.
Using the Advanced Filtering Feature¶
With the above guidelines in mind, let’s try an example. You’ll need to download six files from our Github repo:
- TranscriberFilterComposition.lw
- TranscriberFilter.yaml
- iam_policy.json
- managed_policy.json
- sqs_policy.json
- topic_policy.json
We’ll fugue run the composition to instantiate
infrastructure in the us-west-2
region, then use Transcriber to scan
the region using the YAML file filters. The four JSON files provide IAM
policies used in the composition.
TranscriberFilterComposition.lw
creates 1 DHCP options set, 1 VPC, 2
security groups with 4 rules, 1 Elastic IP address, 2 DynamoDB tables, 2
SQS queues with policy, 2 SNS topics, 1 SNS subscription, 1 IAM policy,
1 IAM managed policy, and 1 IAM role.
Before we get started, change the AWS account number on line 12 of
TranscriberFilterComposition.lw
. Replace the dummy string with your
own AWS account ID.
Next, run the composition and give it the alias filterTest
:
fugue run TranscriberFilterComposition.lw --alias filterTest
When fugue status shows the process has successfully been created, execute the following Transcriber command:
fugue-transcriber --region us-west-2 --filter-file TranscriberFilter.yaml -
This command instructs Transcriber to scan your AWS account in the
us-west-2
region using the filter file TranscriberFilter.yaml
,
then output the results to screen. Note: You can also use -f
to
set the filter file.
You’ll see output like this:
Transcriber starting...
Scanning in us-west-2 region.
Checking for Fugue runtime resources........ not found
Checking for Fugue API server............... not found
Scanning DynamoDB Tables.................... found 2
Scanning EC2 DHCP Options................... found 1
Scanning EC2 Elastic IP Addresses........... found 1
Scanning EC2 Security Groups................ found 2
Scanning EC2 VPCs........................... found 1
Scanning IAM Managed Policies............... found 1
Scanning IAM Roles.......................... found 1
Scanning SNS Subscriptions.................. found 1
Scanning SNS Topics......................... found 2
Scanning SQS Queues......................... found 2
------------------------------------------------------------
Generating Ludwig...
------------------------------------------------------------
[output trimmed for length]
Output trimmed for length.
The output will follow the same format as any other Transcriber run.
Skim the output, and you’ll see that the filters in
TranscriberFilter.yaml
ensure Transcriber returns only the
infrastructure we created with TranscriberFilterComposition.lw
.
Let’s look at the filters more closely. We’ll start with line 8:
- include: aws-ec2-elastic-ip-addresses:*
This is a very simple filter that returns all of the Elastic IP
addresses Transcriber scans in a region. The filter type is include
,
the service to scan is aws-ec2-elastic-ip-addresses
, and the
operator is the wild card *
, so all Elastic IP addresses are listed.
Here’s a more advanced filter on line 2:
- include: aws-ec2-dhcp-options:Tags[].Key == transcriber-test
This filter returns all of the DHCP options sets with a tag key that
matches transcriber-test
. The filter type is include
, the
service to scan is aws-ec2-dhcp-options
, the JMESPath expression is
Tags[].Key
(which flattens the Tags
list and returns each
Key
in the list), the operator is ==
, and the pattern is
transcriber-test
.
Line 16 demonstrates a filter that uses regex:
- include: aws-sns-topics:TopicArn ~= .*:topic[12]
The filter returns only the SNS topics with :topic1
or :topic2
in the ARN. The filter type is include
, the service to scan is
aws-sns-topics
, the JMESPath expression is TopicArn
, the
operator is ~=
, and the regex expression is .*:topic[12]
.
That covers the basics of writing filters. Because the filter file is
written as YAML, you can comment out lines by prepending a #
symbol.
Feel free to comment and uncomment lines to try out different
combinations of filters.
When you’re all done experimenting, just execute
fugue kill filterTest
to terminate the process infrastructure.
Tips for Advanced Filtering¶
- There’s a quick way to find all resources across all services that
match a certain tag name. The convenience value is
tag:name
, and it’s short for*:Tags[?Key=='Name'].Value || TagSet[?Key=='Name'].Value
. Here’s one way to use it:
- include: tag:name == vpc
- Dates are in ISO 8601 format.
- Your filter file can contain both
include
andexclude
filters. Resources accepted by include filters will be passed through exclude filters. If no include filters are specified, all resources will be passed through exclude filters. - Start all filters with a hyphen at the beginning of the line.
- You can filter on reference to a particular resource. This is useful
for filtering on a unique property. Here’s an example that will list
all the instances inside
vpc-76f2ab13
:
- include: aws-ec2-instances:VpcId == vpc-76f2ab13
- Filters can be combined. The following example searches for all
security groups that have a tag key matching
transcriber-test
and have no ingress rules (meaning the security groups have an IP protocol of-1
):
- include:
- aws-ec2-security-groups:Tags[].Key == transcriber-test
- aws-ec2-security-groups:IpPermissionsEgress[].IpProtocol == -1
If you transcribe with this filter while the filterTest
process is
running, Transcriber will return just sg1
, which is the security
group with the description “Empty SG.”
Caveats for Advanced Filtering¶
As we mentioned earlier, advanced filtering is an experimental feature. The list of caveats below is not intended to be exhaustive. If you encounter unexpected behavior, reach out to support@fugue.co with any questions.
You can filter on any AWS field returned in the describe or list call for a Transcriber service. However, some services’ describe and list calls don’t return a lot of info, so other approaches are required:
SQS: The aws sqs list-queues
call returns QueueUrls
URLs and
nothing else. However, once you have a queue URL, you can execute a
second call to return a wider list of properties you can filter on:
aws sqs get-queue-attributes --queue-url QUEUE_URL --attribute-names All
SNS: The aws sns list-topics
call returns TopicArn
ARNs and
nothing else. At this time, that’s the only topic property you can
filter on. You can use regex to return just the topics that match a
given pattern, as we showed in the previous section. You can also use
the ARN to return other resources that refer to it. This filter uses
regex to return all the SNS subscriptions that refer to SNS topics
ending in :topic1
:
- include: aws-sns-subscriptions:TopicArn ~= .*:topic1
A Word of Caution¶
Transcriber is a read-only service, and scanning your account won’t
affect the infrastructure in it. That said, the composition Transcriber
creates through advanced filtering should not be used to launch a
process or import your infrastructure. Advanced filtering is currently
experimental. We cannot guarantee that a composition generated with
advanced filtering will have no side effects when it’s used with
fugue run
or fugue run --import
in production.