Writing Ludwig

Click here to return to the Advanced Ludwig table of contents.

The Ludwig Compiler

Along with the fugue CLI and the Fugue Standard Library, an installation of the Fugue Client Tools comes with the Ludwig compiler, lwc (found in /opt/fugue/bin on Linux and macOS).

It is not necessary to run lwc directly in order to use Fugue. The fugue run command does this for you, compiling whatever Ludwig file you pass it as an argument. Additionally, the fugue run --dry-run command performs compilation and rudimentary runtime validation checks, as well as returning a plan of action from the Conductor, but does not actually run any new processes. If there are compilation or validation errors, they display instead of a plan.

However, these commands do require both a way to reach the SQS queue read by the Conductor for commands, and the time it would take the Conductor to process the request. This might not be suitable for all development purposes, so in such cases the lwc binary can be invoked directly. This requires no network connection whatsoever, and only performs compile-time analysis and validation of compositions. Since the Ludwig compiler runs very quickly, this technique keeps the “feedback loop” for development very short.

The Ludwig compiler has several “backends” that output a variety of formats. The specifics of most of these formats are not important at present (Fugue operators should check that a composition matches their intention using the fugue run --dry-run command), though more details are available at The Ludwig Compiler. By default, lwc uses the null backend, which returns no output if there are no errors or warnings. In certain situations, the simple backend may be more useful; users familiar with JSON and graph data structures should be able to recognize that output as a list of graph node structs.

The lwc --help command should provide helpful information on how to run the compiler. Here are some snippets you can use to set up development with your favorite editor/IDE:

Basic Compilation of Compositions

To quickly compile a composition, use a command like this:

lwc --composition FILE

If no output is returned, the composition has successfully compiled.

Recurring Compilation of Compositions

If you are working with Ludwig in an editor and on a platform with the watch command (available on macOS via Homebrew), you can get IDE-like fast feedback by running the following command in a separate pane or window on-screen:

watch -n2 -- lwc --composition FILE

A similar behavior can be achieved in bash without watch this way:

while true; do clear; lwc --composition FILE; sleep 2; done

Note

Did you know that Fugue offers editor plug-ins so you can read and write Ludwig in your editor/IDE of choice? All plug-ins include Ludwig syntax highlighting, and some have additional features. See ludwig-mode for Emacs, ludwig-vim for vim, vscode-language-ludwig for the Ludwig LSP Server and VSCode, and language-ludwig for Atom.

Ludwig Files, Imports, and the Fugue Standard Library

Ludwig files universally use the extension .lw. This is true regardless of whether the file is a composition or a module; the distinction between these two is in how one uses the file, either directly compiled or imported (using the import statement) from a compiled file, respectively. We discuss the compiler in the next section.

Ludwig imposes no particular structure for file layout of a Ludwig project. A typical project might look like this:

.
├── Lib
│   ├── App.lw
│   └── Vpc.lw
└── MyInfrastructure.lw

The file MyInfrastructure.lw might look like this at any given time:

composition
import Lib.Vpc as VPCs

prodVpc: VPCs.production

The first line instructs the compiler that this file is a Fugue composition, as opposed to a module or other file. A composition file can be run by the fugue CLI.

The second line instructs the compiler to import the contents of ./Lib/Vpc.lw when compiling MyInfrastructure.lw. The way it does this is by searching in folders relative to the compiled file, along the path described by the module name (a similar mechanism exists in Python).

Two items to note:

  • Module names must begin with an uppercase letter.
  • File and path names should match module names, including case. For example, the module Foo.Bar.Baz should have the path and file name Foo/Bar/Baz.lw (not foo/bar/baz.lw).

The fourth line creates a binding, prodVpc, and assigns it the value VPCs.production. In this case, production is a binding in the ./Lib/Vpc.lw file, here aliased (using the as statement) as VPCs.

Explicit Imports

Let’s dig into the concept of imports. By default, when you want to use a library in a composition, you must import the library explicitly. That means you use the import keyword and optionally give the library an alias with the as keyword, as demonstrated in the previous section. Importing a library brings the library and everything inside it in scope, which allows you to use all of its types, constructors, functions, etc.

If you decide not to use an alias for the imported library, you must refer to its types by their fully qualified names, which is the full path to the type in the Fugue Standard Library. Say you imported the EC2 library with the line import Fugue.AWS.EC2, rather than using an alias. If you then wanted to declare a new VPC, you’d call the Vpc.new function by its fully qualified name, Fugue.AWS.EC2.Vpc.new.

Note: Use the fully qualified names from the Fugue.AWS libraries whenever possible, not the lower-level Fugue.Core.AWS libraries. For more information about the different categories of libraries comprising the Fugue Standard Library, skip ahead to this section.

If you give the library an alias, you can refer to its types in Alias.Type format instead of using the type’s fully qualified name. For example, you can import the Fugue.AWS.EC2 library and assign it a much shorter alias of EC2 (or whatever word you like). Instead of typing out Fugue.AWS.EC2.Vpc anytime you want to declare a VPC, you can shorten it to EC2.Vpc.

Here’s an example:

composition

import Fugue.AWS as AWS
import Fugue.AWS.EC2 as EC2

myVpc: EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  region: AWS.Us-west-2
}

As you can see, we’ve used EC2 as an alias for the Fugue.AWS.EC2 library, and we’ve used AWS as an alias for the Fugue.AWS library. That allows us to declare a VPC with the EC2.Vpc.new function instead of Fugue.AWS.EC2.Vpc.new, and declare a region with the AWS.Us-west-2 constructor instead of Fugue.AWS.Us-west-2.

Implicit Imports

Imports are handled explicitly by default. However, Ludwig has an implicit imports mode. When this is enabled, imports are inferred automatically from the source code. That means when you use the fully qualified library name to refer to a type, the compiler will recognize it as a type and follow the path in the library name to import it for you.

To enable implicit imports, include the line @language("implicit-imports") at the very beginning of your composition. The @language pragma helps instruct compiler behavior, and ("implicit-imports") specifies the mode.

If we rewrite the composition in the previous section to implicitly import types, we end up with this:

@language("implicit-imports")

composition

myVpc: Fugue.AWS.EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  region: Fugue.AWS.Us-west-2
}

Since the composition contains the @language line at the beginning, we omit the import lines and refer to types using their fully qualified names. We declare a VPC with the Fugue.AWS.EC2.Vpc.new function, and declare a region with the Fugue.AWS.Us-west-2 constructor.

Note: As with any import, the library you want to import must be in the search path. If you use the fully qualified name Config.Environment.QA, the Config folder must contain the Environment folder, which must contain the QA.lw module.

Explicit vs. Implicit Imports

Ludwig allows you to mix explicit and implicit imports in the same file, in which case the explicit imports take precedence. Let’s modify the composition above and add an explicit import line:

@language("implicit-imports")

composition

import Fugue.AWS as AWS

myVpc: Fugue.AWS.EC2.Vpc.new {
  cidrBlock: "10.0.0.0/16",
  region: AWS.Us-west-2
}

The composition still implicitly imports the Fugue.AWS.EC2.Vpc library, but now it explicitly imports the Fugue.AWS library, aliased AWS. Since explicit imports take precedence, the fully qualified name of Fugue.AWS.Us-west-2 is no longer in scope because it has already been imported as AWS. So, we change it to AWS.Us-west-2.

Deciding whether to use explicit imports, implicit imports, or both is ultimately a matter of personal preference.

If you plan to use both implicit imports and explicit imports, you can forgo the alias (e.g., import Fugue.AWS) and just refer to all types by their fully qualified names. This ensures that each type is in scope no matter how it’s imported.

If you intend to refer to all types by their fully qualified names regardless of import method, it may be more convenient to enable implicit import mode. You can replace all of the explicit import lines with a single @language("implicit-imports") line.

If you find it bothersome to type out the fully qualified library name each time you refer to a type, you might prefer to use the explicit import method and pick a short alias for each library.

Now that we’ve discussed imports, let’s examine the Fugue Standard Library.

The Fugue Standard Library

When you install the Fugue Client Tools on your computer, the Fugue Standard Library is included. The Standard Library is installed in /opt/fugue/lib on Linux and macOS. The Ludwig compiler automatically searches in this path, as well as recursively from your present working directory ($PWD) when you compile.

Every Ludwig composition uses modules from the Fugue Standard Library. For instance, the example seen in Introducing Ludwig By Example would import the Network pattern, using an import statement like this:

import Fugue.AWS.Pattern.Network as Network

There are presently three key categories of module in the Standard Library:

Fugue.AWS.*
These modules provide the tools necessary to concisely express valid infrastructure configurations from basic infrastructure services in AWS. The modules in this library use structures of data similar to those you would see in the AWS API, but with added type information and validity checks. These modules are a good and familiar place to start working with Fugue.
Fugue.AWS.Pattern.*
These modules provide useful infrastructure pattern abstractions. The abstractions here are built by experienced infrastructure professionals here at Fugue, and incorporate good architectural principles and best practices. However, the tools used to build these libraries are the same ones you have, so you can build your own libraries that express your own local best practices and rules.
Fugue.Core.*
These modules are the lowest-level modules from which the preceding ones are built. Any function used in the preceding modules ultimately returns values from these modules at compile time (Ludwig has no “black boxes”). Core values are the ones understood by the Fugue runtime on the Conductor. These modules provide little in the way of guidance or validation, and shouldn’t be used directly, unless services you want to use aren’t found in the preceding modules.

The standard library is documented in the Fugue Standard Library Reference.

Separating Compositions Into Multiple Files

There are lots of reasons you might want to separate a Ludwig composition into multiple files. It can ease merges and collaboration in source control. Separating major infrastructure concerns can help you keep the work straight in your mind and keep the composition more readable. Fortunately, Ludwig supports this.

The key thing to understand is that Fugue only builds things which are top level bindings in a composition, or transitively referred to by such bindings. Values that aren’t in that category are considered extraneous. This is useful for creating libraries, as one doesn’t automatically want to use everything that is in a particular library, but it does mean that multi-file compositions must be structured so that all resources stem from a single composition file.

The best way to do this is to create a master composition file that serves as an entry point for the composition, and then stitches together the constituent files in a little bit of code. Consider the following trivial composition that builds an SQS queue as well as an SNS topic.

Separation.lw

This file serves as our entry point. It has the composition keyword, and imports the constituent files.

composition

import Separation_SNS
import Separation_SQS

sns: Separation_SNS.resources
sqs: Separation_SQS.resources

Note the bindings sns and sqs. These bindings refer to a value in the constituent files. This value is a manually created tuple that contains all the resources from that file you wish to include in the composition.

Separation_SNS.lw

This file encompasses the SNS concerns in this composition.

import Fugue.AWS as AWS
import Fugue.Core.AWS.SNS as SNS

resources: (
  my-topic
)

my-topic: SNS.Topic{
  name: "my-topic",
  region: AWS.Us-east-1
}

The binding my-topic describes the topic (as you might expect), and the binding resources is a tuple that includes my-topic. This is, in turn, the resources that Separation.lw is referring to in the sns binding.

You might note that we could just as easily bind sns to Separation_SNS.my-topic, and you’d be correct about that. However, a collection like this is quite handy when a file declares more than one resource, which is quite common. This way, the entry point file doesn’t have to change if, for example, an SNS topic subscription is added – it still just binds to resources.

Separation_SQS.lw

This file encompasses the SQS concerns in this composition. It follows the exact same pattern as the SNS file.

import Fugue.AWS as AWS
import Fugue.AWS.SQS as SQS

resources: (
  my-queue
)

my-queue: SQS.Queue.new {
  name: "my-queue",
  region: AWS.Us-west-2,
  maximumMessageSize: 1024,
  messageRetentionPeriod: 120,
  visibilityTimeout: 0,
}

Using this technique, you can easily compose your compositions from multiple files, either authored by you or others in the Fugue community.

Referring to External Resources

Sometimes you might have resources that are managed outside of Fugue, such as things created manually in the AWS console. Fugue includes the ability to refer to these external resources with the external function.

When used in a Ludwig composition, external designates a resource as non-Fugue managed. Depending on how it’s implemented, this designation can instruct Fugue to exempt the resource(s) from inspection by Ludwig and excludes them from type-checking or validation.

Defining External

Fugue defines external resources as any non-Fugue managed resources. Fugue recognizes a variety of AWS resources typically via an ID, Name, or ARN (for information on ARNs, refer to this). Depending on the service, Fugue returns a Ludwig reference to the service of the same type as a Fugue-managed reference to that service.

Note: Not all services are covered and the full list of supported services for this functionality are expected to evolve with our AWS service coverage. Reach out to support@fugue.co with any questions or issues.

Using External Resources

The external function uses an identifier, or a string prefix, that allows us to perform type-checking and validation.

In the following example the resourceID needs to start with an identifier, in this case a VPC, so the resourceID uses the format vpc-1234abcd.

Usage and Examples

EC2.Vpc Example

Rather than creating a new VPC using the EC2.Vpc.new {} constructor, you can use this line of Ludwig to point to an external VPC. Just replace vpc-1234abcd with the proper resource ID, and AWS.Us-east-1 with the proper region.

vpc: EC2.Vpc.external("vpc-1234abcd", AWS.Us-east-1)

SNS.Topic Example

Likewise, you can use this line of Ludwig to reference an externally-managed SNS topic instead of creating a new topic with the SNS.Topic.new {} constructor. Replace the ARN here with the actual ARN of your topic.

topic: SNS.Topic.external("arn:aws:sns:us-east-1:123456789012:example-topic")

Lambda.Function Example

And you can use this line of Ludwig to reference an externally-managed Lambda function instead of using the Lambda.Function.new {} constructor to create a new function. Again, replace the ARN with the actual ARN of your Lambda function.

myLambda: Lambda.Function.external("arn:aws:lambda:us-east-1:123456789012:example-function")

Resources That Don’t Support Externals

Certain resources do not support externals. For example, the EC2.Subnet type cannot refer to an external VPC, and an AutoScaling.ScalingPolicy cannot refer to an external Auto Scaling group. Fugue provides a runtime validation and design-time validation to check compositions for unsupported external references. (Full list appears below.)

Runtime Validation

The Conductor ships with a validation module (Fugue.AWS.Externals.lw) that raises an error at runtime if a composition includes a reference to an unsupported external. This validation is automatically enabled and cannot be disabled.

For example, if your composition includes this binding:

# This binding won't compile because the EC2.Subnet references EC2.Vpc.external
mySubnet: EC2.Subnet.new {
  availabilityZone: AWS.A,
  cidrBlock: "10.0.0.0/24",
  vpc: EC2.Vpc.external("vpc-1234abcd", AWS.Us-west-2),
}

You’ll see this validation error when you run the composition:

lwc Test.lw --validation-modules Fugue.AWS.Externals
ludwig (validation error):
  "/opt/fugue/lib/Fugue/AWS/EC2/Subnet.lw" (line 86, column 15):
  Validations failed:

    86|     resource: EC2.Subnet {
    87|                 vpc: vpc,
    88|                 cidrBlock: cidrBlock,
    89|                 availabilityZone: availabilityZone,
    90|                 mapPublicIpOnLaunch: mapPublicIpOnLaunch,
    91|                 tags: tags,
    92|               }

    - Fugue Conductor does not support external vpc in Fugue.Core.AWS.EC2.Subnet
      (from Fugue.AWS.Externals.limitEC2Subnet)

  Stack trace:
    In call to new at "MyComposition.lw" (line 6, column 11)

Design-Time Validation

The validation module is also included in the Fugue Standard Library, a component of the Fugue Client Tools. To optionally validate your composition at design-time, use the Ludwig compiler’s --validation-modules option to specify the validation module:

lwc MyComposition.lw --validation-modules Fugue.AWS.Externals

To see a list of all the validations for unsupported externals, see Fugue.AWS.Externals or view the Fugue.AWS.Externals.lw file on your computer:

cat /opt/fugue/lib/Fugue/AWS/Externals.lw

List of Unsupported Externals

A New Way of Specifying EBS Volumes on EC2 Instances

In order to enhance EBS support, a new method of specifying EBS volumes on EC2 instances has been introduced, along with new types and new Ludwig libraries.

This corrects known issues such as this one: “If an instance is launched into one Availability Zone, and an EBS Volume is defined to be attached to that instance, but in a different Availability Zone, the composition will pass compilation, but the Volume will be created in the same Availability Zone as the Instance.”

It also improves the enforcement and mutability of io1 volume input/output operations per second (IOPS). Because io1 is a Provisioned IOPS volume type, Fugue enforces the IOPS rate you specify and allows you to change that attribute without doing a destructive update. All volume tags are likewise mutable and enforced. For more information, see EC2.Volume.

The old method of declaring EBS volumes within EC2 instances is deprecated, but Fugue honors it for backwards compatibility. Your composition may contain EBS declarations in the new style or the old style; it cannot contain both.

Note: We do not recommend updating a process from the new style back to the old style. It will result in a destructive update.

A method of easily migrating processes from the old style to the new style is forthcoming. In the meantime, reach out to support@fugue.co with any questions.

The Old Style

This is an example of how EBS volumes were formerly declared within EC2 instances:

myOldStyleRootVolume: EC2.Volume.new {
  availabilityZone: AWS.A,
  size: 16,
}

myOldStyleVolume: EC2.Volume.new {
  availabilityZone: AWS.A,
  size: 24,
}

myOldStyleInstance: EC2.Instance.new {
  instanceType: EC2.T2_micro,
  subnet: mySubnet,
  blockDeviceMappings: [
    EC2.InstanceBlockDeviceMapping {
      deviceName: "/dev/xvda",
      ebs: EC2.EbsInstanceBlockDevice {
        volume: myOldStyleRootVolume
      },
    },
    EC2.InstanceBlockDeviceMapping {
      deviceName: "/dev/xvdb",
      ebs: EC2.EbsInstanceBlockDevice {
        volume: myOldStyleVolume
      },
    },
    EC2.InstanceBlockDeviceMapping {
      deviceName: "/dev/xvdc",
      virtualName: "ephemeral0",
    },
  ],
  securityGroups: [mySecurityGroup],
  image: "ami-e689729e", # Amazon Linux
}

The New Style

Here’s how EBS volumes are configured for EC2 instances now:

myNewStyleRootVolume: EC2.RootBlockDevice.new {
  volumeSize: 16,
}

myNewStyleVolume: EC2.Volume.new {
  availabilityZone: AWS.A,
  region: AWS.Us-west-2,
  size: 24,
}

myNewStyleInstance: EC2.Instance.new {
  instanceType: EC2.M3_large,
  subnet: mySubnet,
  securityGroups: [mySecurityGroup],
  image: "ami-e689729e", # Amazon Linux
  rootBlockDevice: myNewStyleRootVolume,
  volumes: [
    EC2.VolumeAttachment.new {
      volume: myNewStyleVolume,
      deviceName: "/dev/xvdb"
    }
  ],
  instanceStores: [
    EC2.InstanceStore.new {
      deviceName: "/dev/xvdc",
      virtualName: "ephemeral0"
    }
  ],
}

Summary of Changes

These are the major changes:

If you have any questions, reach out to support@fugue.co.