Logging & Notifications

CLI Logging

The Fugue CLI keeps a running log of all CLI activity, including commands entered, API calls sent, errors encountered, and output printed to the screen. The log also tracks configuration information. The log file, fuguecli.log, is created in the same directory in which the Fugue CLI is run. This log is helpful for troubleshooting.

Small selection of entries logged while executing a suspend command:

2018-02-07T20:53:01.02 [ fugue ] DEBUG - environment lookup failed 'FUGUE_USER_PROFILE', attempting config lookup
2018-02-07T20:53:01.02 [ fugue ] DEBUG - attempting lookup for 'region' by environment 'FUGUE_CONDUCTOR_REGION'
2018-02-07T20:53:01.02 [ fugue ] DEBUG - environment lookup successful FUGUE_CONDUCTOR_REGION=us-east-1
2018-02-07T20:53:01.02 [ fugue ] INFO - using default profile default-141874191075-us-east-1
2018-02-07T20:53:01.02 [ fugue ] DEBUG - Loading profile 'default-141874191075-us-east-1'
2018-02-07T20:53:01.02 [ fugue ] DEBUG - found profile default-141874191075-us-east-1 in /Users/main-user/projects/credentials
2018-02-07T20:53:01.02 [ fugue ] INFO - retrieved secret from creds file: /Users/main-user/projects/credentials
2018-02-07T20:53:01.02 [ fugue ] DEBUG - attempting lookup for 'requestQueue' by environment 'FUGUE_CONDUCTOR_REQUESTQUEUE'
2018-02-07T20:53:01.02 [ fugue ] DEBUG - environment lookup failed 'FUGUE_CONDUCTOR_REQUESTQUEUE', attempting config lookup
2018-02-07T20:53:01.02 [ fugue ] DEBUG - attempting lookup for 'responseTable' by environment 'FUGUE_CONDUCTOR_RESPONSETABLE'
2018-02-07T20:53:01.02 [ fugue ] DEBUG - environment lookup failed 'FUGUE_CONDUCTOR_RESPONSETABLE', attempting config lookup
2018-02-07T20:53:02.02 [ demarccore.endpoints ] INFO - SQS Message Sent: 4d8afe1f-cea5-4132-8f83-1abed15b9c5b
2018-02-07T20:53:03.02 [ demarccore.endpoints ] INFO - SQS Message Sent: 6f1186ac-b642-400f-a171-c004ee528d64
2018-02-07T20:53:04.02 [ fugue.screen ] INFO - Requesting the Conductor to suspend process ...
2018-02-07T20:53:04.02 [ demarccore.endpoints ] INFO - SQS Message Sent: bdbe6722-e27c-4fb1-bbfa-c190cc01ba13
2018-02-07T20:53:06.02 [ fugue.screen ] INFO - [ DONE ] Process with alias: hello is being suspended.
2018-02-07T20:53:06.02 [ fugue.screen ] INFO - [ HELP ] Run the 'fugue status' command to view details and status of this process suspension.

The full set of log entries for the above command also includes a section called “Configuration Info” that lists debug-level details about the CLI version, the fugue.yaml file path if present, the Conductor region, and Conductor resource names (S3 bucket, SQS queues). This is followed by a series of internal messages related to carrying out the suspend command, ending with user-facing messages output to the screen.

For information on using the CLI log to facilitate debugging, see Troubleshooting (Fugue Commands).

CloudWatch Logs

At present, the best way to find out what is going on in Fugue is to use the CLI‘s status or history command or read the CLI log. However, you can also get a look at the logs for Fugue through the AWS CloudWatch logs service. This chapter will give you some useful guidance on how to find relevant logs, although it remains the case that they are somewhat technical and opaque for now.


Want to change the verbosity of Conductor component logs? See the fugue runtime command.

Fugue CloudWatch Log Format

Fugue logs are all output in JSON serialization format with events delineated by line breaks. The log file itself creates an LDJSON stream.

Each log event contains the following information:

Field Details
timestamp Displays the timestamp information in UTC.
component Identifies the name of the component.
log_level Provides a severity level from 0 to 7.
message Contains the primary message for the log entry.
error_detail (optional) In the event of an error, this field provides diagnostic details.
fid (if applicable) Reports the associated Fugue process ID.
job_id (if applicable) Reports the associated component-assigned job ID.

How to Find Fugue Logs in CloudWatch

Most relevant log data for any details you want to find about Fugue are found in the /fugue/conductor log group in CloudWatch.

The ~/fugue/conductor~ log group.

The /fugue/conductor log group.

The two most relevant logs are those for the Manager, which controls the planning components of Fugue, and the Broker, which interacts with infrastructure provider APIs. These can each be found, respectively, in the manager and fugue-broker log streams.

The ~fugue-broker~ log stream.

The fugue-broker log stream.

In either case, the most valuable thing to do is to filter by FID. The FID is the “Fugue ID” of the process, and is returned to you after a fugue run, or in a fugue status command.

As an example, here is a filter that can be applied to either log: { $.fid = "c0bc1b09-c0c1-403a-bcee-d3f56bba8741" } Of course, you’ll need to substitute your own subject FID between the quotes.

Common Log Messages and Patterns

Here are a few message types to look for to help you find out what’s going on with Fugue. For now, we’ll just focus on looking at processes that are in the Running state.

Manager Log

Planned Actions

When viewing the Manager log filtered by FID as shown above, you can look at actions Fugue has planned to take. These won’t differ much from what you see in the Broker, although you can get a good idea of how Fugue “thinks” by tracking planned instructions in this log.

  "account_id": "",
  "component": "manager",
  "fid": "04cf823e-1217-4aaf-b220-697ab4c0ac84",
  "guid": "04cf823e-1217-4aaf-b220-697ab4c0ac84.3db9a301-bd1d-55e7-96d5-80804348ab63",
  "job_id": "1465231683",
  "layer": "emit-instructions",
  "log_level": "debug",
  "message": "Command aws.ec2.create_vpc for resource 04cf823e-1217-4aaf-b220-697ab4c0ac84.3db9a301-bd1d-55e7-96d5-80804348ab63 in account ID  region us-west-2 added by go-planner on layer emit-instructions",
  "params": "{\"CidrBlock\":\"\",\"InstanceTenancy\":\"default\"}",
  "planner": "go-planner",
  "region": "us-west-2",
  "request_type": "aws.ec2.create_vpc",
  "timestamp": "2016-06-06T16:48:08.637564"

Note the message field. The plan is generally a sequence of “commands,” so describes one step in the plan.

Broker Log

API Requests

When you’re looking at the Broker log filtered by FID, you should see lots of messages like this if you ran a composition that defines any infrastructure:

  "timestamp": "2016-06-06T16:48:11.592",
  "component": "broker.job",
  "log_level": "INFO",
  "message": "Issuing [<botocore.client.EC2 object at 0x7f8088359358>.create_vpc] with [{'InstanceTenancy': 'default', 'CidrBlock': ''}]",
  "fid": "04cf823e-1217-4aaf-b220-697ab4c0ac84",
  "job_id": "1465231683"

Note the message field. A message like this (Issuing...) indicates specific API calls that Fugue is making to AWS.

Audit Log

In addition to the Broker logs, Fugue also retains an audit log. While some of the information within the audit logs also appear in the Broker logs, the audit logs are much easier to parse. Audit logs contain two main pieces of information.

The first are executed lines logged when Fugue runs a command in response to a specific user action. These lines also contain details about the Fugue user that started the action.

For example:

Mar 12 20:37:58 ip-10-0-3-212 /fugue_broker: Fugue executed [delete_role] with input [{'RoleName': 'example-role'}] and result [
] for user [root] in account [fugue-1234567891098]

The second type are lines logged when Fugue runs a command as part of an enforcement action.

For example:

Mar 12 20:39:58 ip-10-0-3-212 /fugue_broker: Fugue automatically executed [put_bucket_policy] with input [{'Bucket': 'example-bucket', 'Policy': '{\n "Version": "2012-10-17",\n "Statement": [\n {\n "Sid": "AWSCloudTrailAclCheck20150319",\n "Effect": "Allow",\n "Principal": {"Service": "cloudtrail.amazonaws.com"},\n "Action": "s3:GetBucketAcl",\n "Resource": [\n "arn:aws:s3:::example-bucket"\n ]\n },\n {\n "Sid": "AWSCloudTrailWrite20150319",\n "Effect": "Allow",\n "Principal": {"Service": "cloudtrail.amazonaws.com"},\n "Action": "s3:PutObject",\n "Resource": "arn:aws:s3:::example-bucket/*",\n "Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}\n }\n ]\n}'}] and result [
] in account [fugue-1234567891098] in response to an external change

Metrics and Alarms

Performance metrics

By default, Fugue provides two general types of metrics, both of which are visible under Metrics in the CloudWatch section of the AWS Console. The first type of metrics is general peformance metrics for components of the Conductor, available under the Custom Metrics section.

Custom Fugue metrics

Custom Fugue metrics.

Note: Fugue has the ability to support additional metrics; however, these are typically only enabled as part of troubleshooting. Contact us with any questions - support@fugue.co!

Health Checks and Alarms

The Fugue Conductor’s internal health checker is a component that monitors the health of all other Conductor components. If any components go down or report issues, these details are sent via logs to CloudWatch and trigger any necessary alarms.

This internal health checker monitors two specific statuses for the Conductor:

  • if the Conductor is alive and reporting data, and;
  • if the Conductor (and all components) are healthy.

All status information, alarms, and logs related to this monitoring are viewable through the AWS Console.

Conductor Alive

The Conductor alive function tracks the Conductor’s internal health checker to determine that it is up and reporting data (i.e., value = 0 is OK and value = x is INSUFFICIENT_DATA). If the internal health checker fails to report data for more than 3 minutes, a notification is triggered to update the status from OK to INSUFFICIENT_DATA. Unlike the component health checker, the alive function monitors data reporting and is either receiving data (value = 0, alive, OK) or is not receiving data (value = x, no data, INSUFFICIENT_DATA).

Conductor Health Check

The Conductor’s internal health checker is also designed to perform regular health checks and provide metrics on individual components. The CloudWatch alarm is not currently separated into individual alarms for each component; instead, when “healthy,” the AWS Console reports instances as OK. If issues arise with any Conductor component, the internal health checker triggers a CloudWatch alarm with the status ALARM.

  • An alarm triggers when an instance reports with a value = 1 that persists for more than 3 minutes (value = 0 is healthy)
  • Details about the specifics of the alarm are viewable in the CloudWatch section of the AWS Console in both the Alarms section and the Logs section

Note: the AWS CloudWatch console contains all of the data and logs related to any potential issues with Fugue or the Conductor; however, due to the level of detail and complexity we recommend that you reach out to support@fugue.co for assistance with any troubleshooting.

The log stream alarms

Fugue Conductor alarms.

Event Notifications

Fugue supports subscriber-based activity notifications. By subscribing to a particular SNS topic through the AWS Management Console, you can receive emails when Fugue takes certain activity. The emails include pertinent details and a full plan in JSON format. You also have the option to receive the email output entirely in JSON to make it easier to parse for use in external tools.

Currently supported activities are:

  • Kill - when the user executes kill
  • Resume - when the user executes resume
  • Run - when the user executes run
  • Suspend - when the user executes suspend
  • System - when Fugue executes a system job (notification is only sent on a job that failed or if a process is halted)
  • Update - when the user executes update
  • Drift - when Fugue executes a job to correct drift (infrastructure changes that didn’t result from a user command)
  • Release - when a user executes release on a Fugue process

Each activity corresponds to an SNS topic prefaced with fugue-notifications- (for example, fugue-notifications-kill, or fugue-notifications-kill-json).

How To Sign Up

With the AWS Console

Head to the SNS Dashboard in the region in which the Conductor is running, then select “Topics” in the left sidebar.

Select a topic beginning with fugue-notifications-, then click “Subscribe to topic” in the “Action” drop-down menu. To receive JSON notifications choose the topic that includes -json, for example fugue-notifications-kill-json.

In the box that pops up, change the protocol from “HTTP” to “Email.” Enter your email address as an endpoint and click “Create Subscription.” Note: Currently, email is the only supported protocol.

Check your email for a subscription confirmation, and confirm your subscription by clicking on the link.

With the AWS CLI

Run the following command, replacing <region> with your Conductor’s region, <account> with your AWS account ID, <topic_name> with the proper fugue-notifications- action (and optionally include -json for JSON notifications), and <email_addr> with your email address:

aws sns subscribe --topic-arn arn:aws:sns:<region>:<account>:<topic_name> \
                      --protocol email \
                      --notification-endpoint <email_addr>

Check your email for a subscription confirmation, and confirm your subscription by clicking on the link.

Sample Email

Here is a sample email received after executing update on a running composition, changing a VPC’s tags:

Fugue has taken action on something you might be interested in.

Job Type: UPDATE
Job ID: 1496956416
Account: fugue-xxxxxxxxxxxx
Process FID: f1e2f648-39e4-4934-8e8f-0a8f0f33b193
Correlation ID: 5c4ff870-b304-4f45-af41-d437a020a78d

    "fid": "f1e2f648-39e4-4934-8e8f-0a8f0f33b193",
    "job_id": "1496956416",
    "requests": [
            "guid": "f1e2f648-39e4-4934-8e8f-0a8f0f33b193.f6ba84b3-b220-5ed5-b49c-76a7c4890187.tags",
            "params": {
                "Resources": [
                "Tags": [
                        "Key": "Application",
                        "Value": "Hello World2"
            "region": "us-west-2",
            "request_type": "aws.ec2.create_tags"
            "guid": "f1e2f648-39e4-4934-8e8f-0a8f0f33b193.a8106f57-7525-5c0c-8af6-206ee3ff1058.tags",
            "params": {
                "Resources": [
                "Tags": [
                        "Key": "Application",
                        "Value": "Hello World2"
            "region": "us-west-2",
            "request_type": "aws.ec2.create_tags"

How To Unsubscribe

To unsubscribe from any SNS notification, click on the “Unsubscribe” link at the bottom of the email, delete the subscription through the SNS Dashboard in the AWS Management Console, or use the following AWS CLI command:

aws sns unsubscribe --subscription-arn <subscription_arn>