Writing Ludwig

The Ludwig Compiler

Along with the fugue CLI and the Fugue Standard Library, an installation of the Fugue Client Tools comes with the Ludwig compiler, lwc (found in /opt/fugue/bin on Linux and OSX).

It is not necessary to run lwc directly in order to use Fugue. The fugue run command does this for you, compiling whatever Ludwig file you pass it as an argument. Additionally, the fugue run --dry-run command performs compilation and rudimentary runtime validation checks, as well as returning a plan of action from the Conductor, but does not actually run any new processes. If there are compilation or validation errors, they display instead of a plan.

However, these commands do require both a way to reach the SQS queue read by the Conductor for commands, and the time it would take the Conductor to process the request. This might not be suitable for all development purposes, so in such cases the lwc binary can be invoked directly. This requires no network connection whatsoever, and only performs compile-time analysis and validation of compositions. Since the Ludwig compiler runs very quickly, this technique keeps the “feedback loop” for development very short.

The Ludwig compiler has several “backends” that output a variety of formats. The specifics of most of these formats are not important at present (Fugue operators should check that a composition matches their intention using the fugue run --dry-run command), though more details are available at The Ludwig Compiler. By default, lwc uses the null backend, which returns no output if there are no errors or warnings. In certain situations, the simple backend may be more useful; users familiar with JSON and graph data structures should be able to recognize that output as a list of graph node structs.

The lwc --help command should provide helpful information on how to run the compiler. Here are some snippets you can use to set up development with your favorite editor/IDE:

Basic Compilation of Compositions

To quickly compile a composition, use a command like this:

lwc --composition FILE

If no output is returned, the composition has successfully compiled.

Recurring Compilation of Compositions

If you are working with Ludwig in an editor and on a platform with the watch command (available on OSX via Homebrew), you can get IDE-like fast feedback by running the following command in a separate pane or window on-screen:

watch -n2 -- lwc --composition FILE

A similar behavior can be achieved in bash without watch this way:

while true; do clear; lwc --composition FILE; sleep 2; done

Note

Did you know that Fugue offers editor plug-ins so you can read and write Ludwig in your editor/IDE of choice? All plug-ins include Ludwig syntax highlighting, and some have additional features. See ludwig-mode for Emacs, ludwig-vim for vim, vscode-language-ludwig for VSCode, and language-ludwig for Atom.

Ludwig Files, Imports, and the Fugue Standard Library

Ludwig files universally use the extension .lw. This is true regardless of whether the file is a composition or a library; the distinction between these two is in how one uses the file, either directly compiled or imported (using the import statement) from a compiled file, respectively. We discuss the compiler in the next section.

Ludwig imposes no particular structure for file layout of a Ludwig project. A typical project might look like this:

.
├── Lib
│   ├── App.lw
│   └── Vpc.lw
└── MyInfrastructure.lw

The file MyInfrastructure.lw might look like this at any given time:

composition
import Lib.Vpc as VPCs

prodVpc: VPCs.production

The first line instructs the compiler that this file is a Fugue composition, as opposed to a library or other file. A composition file can be run by the fugue CLI.

The second line instructs the compiler to import the contents of ./Lib/Vpc.lw when compiling MyInfrastructure.lw. The way it does this is by searching in folders relative to the compiled file, along the path described by the module name (a similar mechanism exists in Python). Note that module names must begin with an uppercase letter.

The fourth line creates a binding, prodVpc, and assigns it the value VPCs.production. In this case, production is a binding in the ./Lib/Vpc.lw file, here aliased (using the as statement) as VPCs.

The Fugue Standard Library

When you install the Fugue Client Tools on your computer, the Fugue Standard Library is included. The Standard Library is installed in /opt/fugue/lib on Linux and OSX. The Ludwig compiler automatically searches in this path, as well as recursively from your present working directory ($PWD) when you compile.

Every Ludwig composition uses modules from the Fugue Standard Library. For instance, the example seen in Introducing Ludwig By Example would import the Network pattern, using an import statement like this:

import Fugue.AWS.Pattern.Network as Network

There are presently three key categories of module in the Standard Library:

Fugue.AWS.*
These modules provide the tools necessary to concisely express valid infrastructure configurations from basic infrastructure services in AWS. The modules in this library use structures of data similar to those you would see in the AWS API, but with added type information and validity checks. These modules are a good and familiar place to start working with Fugue.
Fugue.AWS.Pattern.*
These modules provide useful infrastructure pattern abstractions. The abstractions here are built by experienced infrastructure professionals here at Fugue, and incorporate good architectural principles and best practices. However, the tools used to build these libraries are the same ones you have, so you can build your own libraries that express your own local best practices and rules.
Fugue.Core.*
These modules are the lowest-level modules from which the preceding ones are built. Any function used in the preceding modules ultimately returns values from these modules at compile time (Ludwig has no “black boxes”). Core values are the ones understood by the Fugue runtime on the Conductor. These modules provide little in the way of guidance or validation, and shouldn’t be used directly, unless services you want to use aren’t found in the preceding modules.

The standard library is documented in the Fugue Standard Library Reference.

Separating Compositions Into Multiple Files

There are lots of reasons you might want to separate a Ludwig composition into multiple files. It can ease merges and collaboration in source control. Separating major infrastructure concerns can help you keep the work straight in your mind and keep the composition more readable. Fortunately, Ludwig supports this.

The key thing to understand is that Fugue only builds things which are top level bindings in a composition, or transitively referred to by such bindings. Values that aren’t in that category are considered extraneous. This is useful for creating libraries, as one doesn’t automatically want to use everything that is in a particular library, but it does mean that multi-file compositions must be structured so that all resources stem from a single composition file.

The best way to do this is to create a master composition file that serves as an entry point for the composition, and then stitches together the constituent files in a little bit of code. Consider the following trivial composition that builds an SQS queue as well as an SNS topic.

Separation.lw

This file serves as our entry point. It has the composition keyword, and imports the constituent files.

composition

import Separation_SNS
import Separation_SQS

sns: Separation_SNS.resources
sqs: Separation_SQS.resources

Note the bindings sns and sqs. These bindings refer to a value in the constituent files. This value is a manually created tuple that contains all the resources from that file you wish to include in the composition.

Separation_SNS.lw

This file encompasses the SNS concerns in this composition.

import Fugue.AWS as AWS
import Fugue.Core.AWS.SNS as SNS

resources: (
  my-topic
)

my-topic: SNS.Topic{
  name: "my-topic",
  region: AWS.Us-east-1
}

The binding my-topic describes the topic (as you might expect), and the binding resources is a tuple that includes my-topic. This is, in turn, the resources that Separation.lw is referring to in the sns binding.

You might note that we could just as easily bind sns to Separation_SNS.my-topic, and you’d be correct about that. However, a collection like this is quite handy when a file declares more than one resource, which is quite common. This way, the entry point file doesn’t have to change if, for example, an SNS topic subscription is added – it still just binds to resources.

Separation_SQS.lw

This file encompasses the SQS concerns in this composition. It follows the exact same pattern as the SNS file.

import Fugue.AWS as AWS
import Fugue.AWS.SQS as SQS

resources: (
  my-queue
)

my-queue: SQS.Queue.new {
  name: "my-queue",
  region: AWS.Us-west-2,
  maximumMessageSize: 1024,
  messageRetentionPeriod: 120,
  visibilityTimeout: 0,
}

Using this technique, you can easily compose your compositions from multiple files, either authored by you or others in the Fugue community.

Referring to External Resources

Sometimes you might have resources that are managed outside of Fugue, such as things created manually in the AWS console. Fugue includes the ability to refer to these external resources with the external function.

When used in a Ludwig composition, external designates a resource as non-Fugue managed. Depending on how it’s implemented, this designation can instruct Fugue to exempt the resource(s) from inspection by Ludwig and excludes them from type-checking or validation.

Defining External

Fugue defines external resources as any non-Fugue managed resources. Fugue recognizes a variety of AWS resources typically via an ID, Name, or ARN (for information on ARNs, refer to this). Depending on the service, Fugue returns a Ludwig reference to the service of the same type as a Fugue-managed reference to that service.

Note: Not all services are covered and the full list of supported services for this functionality are expected to evolve with our AWS service coverage. Reach out to support@fugue.co with any questions or issues.

Using External Resources

The external function uses an identifier, or a string prefix, that allows us to perform type-checking and validation.

In the following example the resourceID needs to start with an identifier, in this case a VPC, so the resourceID uses the format vpc-1234abcd.

Usage and Examples

EC2.Vpc Example

Rather than creating a new VPC using the EC2.Vpc.new {} constructor, you can use this line of Ludwig to point to an external VPC. Just replace vpc-1234abcd with the proper resource ID, and AWS.Us-east-1 with the proper region.

vpc: EC2.Vpc.external("vpc-1234abcd", AWS.Us-east-1)

SNS.Topic Example

Likewise, you can use this line of Ludwig to reference an externally-managed SNS topic instead of creating a new topic with the SNS.Topic.new {} constructor. Replace the ARN here with the actual ARN of your topic.

topic: SNS.Topic.external("arn:aws:sns:us-east-1:123456789012:example-topic")

Lambda.Function Example

And you can use this line of Ludwig to reference an externally-managed Lambda function instead of using the Lambda.Function.new {} constructor to create a new function. Again, replace the ARN with the actual ARN of your Lambda function.

myLambda: Lambda.Function.external("arn:aws:lambda:us-east-1:123456789012:example-function")