Using fregot repl to Debug Custom Rules

Note

Don’t forget to download the fugue.rego library and retrieve a test input file!

This document focuses on using Fregot’s repl command to test and debug custom rules. The REPL is useful for debugging a rule and evaluating queries inside a rule; it is ideal for both simple rules and advanced rules.

Before starting up the REPL

There are a few important prerequisites before you jump inside the REPL:

1. Download the fugue.rego library and retrieve a test input file

2. Add a package declaration and import the fugue library inside your rule file (see instructions)

3. Convert Fugue resource names to Terraform resource names inside advanced rules (see instructions)

Use Terraform resource names in advanced rules

Before you hop into the REPL, there’s one change you need to make to the rule if it’s an advanced rule: you need to replace the Fugue name for the resource type with the Terraform name in the fugue.resources(resource_type) call(s). This is because Fregot is a general-purpose Rego tool and is cleanly separated from Fugue custom rules. To find the Terraform name, open up the input JSON document and look for the _type nested property under the desired resource:

{
  "resources": {
    "azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p": {
      "_provider": "provider.azurerm",
      "_skeleton": {
        "depends_on": null,
        "deposed": null,
<--cut for brevity-->
      },
      "_type": "azurerm_virtual_machine",
<--cut for brevity-->

You can also refer to the Fugue service coverage documentation for AWS, Azure, and Google.

Given the input above, you would change fugue.resources("Azure.Compute.VirtualMachine") to fugue.resources(azurerm_virtual_machine) in the Rego rule file.

Why is this only necessary with advanced rules? That’s because the resource type is specified externally in simple rules – not inside the .rego file with a fugue.resources(resource_type) call. When you test a simple rule, you test one resource at a time, so Fregot determines the resource type automatically based on what’s in the input document.

Tip

Though Fregot doesn’t recognize Fugue resource names, Fugue recognizes Terraform resource names. So, no need to switch back to the Fugue name — you can keep the Terraform name if you like!

Launch the REPL

To launch Fregot’s REPL, use the following command, replacing the filenames with your own filenames:

fregot repl <rule filename> fugue.rego --input <JSON input filename> --watch

For example:

fregot repl vm_size.rego fugue.rego --input input.json --watch

This command instructs Fregot to load the rule and Fugue library, set the input document, and enable the watch feature. (You can tell Fregot to load multiple rules files at once! Just list each filename consecutively, like vm_size.rego azure_vms_portal_diagnostics.rego path/to/a_third_rule.rego fugue.rego etc.)

You’ll see that you’re inside the REPL:

F u g u e   R E G O   T o o l k i t
fregot v0.14.2 repl - use :help for usage info
repl%

Now, open the rule package you’d like to test. If you loaded the file already by launching the REPL with the filename specified, as we did above, you can use the :open command with the package name to open it:

:open <rule package name>

This is handy if you’ve loaded multiple rule files and you want to switch between them.

Alternatively, you can use the :load command with the filename (not package name) to load and open a file all at once:

:load <rule filename>

Once you’re inside a custom rule file, you can evaluate Rego rules within the file by simply entering their names.

Evaluate the allow, deny, or policy rule

The steps you take to test a custom rule with Fregot vary slightly depending on whether it’s a simple or advanced rule.

Simple rules

Simple rules operate on a single resource at a time and advanced rules operate on an entire environment at a time. To test a simple rule, you tell Fregot exactly which resource to test, using the resource’s identifier in double-quotes:

allow with input as input.resources["<resource identifier>"]

For example:

allow with input as input.resources["aws_db_instance.xyz123abc456xyz123abc456zyx1234"]

You can find the resource identifier in the input document:

{
  "resources": {
    "aws_db_instance.xyz123abc456xyz123abc456zyx1234": {
      "_provider": "provider.aws.us-east-1",
      "_skeleton": {
<--cut for brevity-->

If allow or deny is true, Fregot outputs a boolean value:

= true

If allow is true, the resource is compliant; if deny is true, the resource is noncompliant. Otherwise, Fregot returns no results (i.e., an empty set, {}), or an explicit false if your rule includes the line default deny = false or default allow = false.

For more about the resource identifier vs. ID, see this note.

Advanced rules

Unlike simple rules, advanced rules don’t require you to specify a particular resource during testing. If you’ve passed in the input document using --input when launching the REPL or :input <input filename> inside the REPL, Fregot looks at all the resources specified using fugue.resources(resource_type). That means you can simply evaluate the policy rule as shown below:

policy

You’ll see the policy set containing an object for each evaluated resource:

= {
  {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/windowsvm",
    "valid": false,
    "type": "azurerm_virtual_machine",
    "message": "invalid"
  },
  {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
    "valid": true,
    "type": "azurerm_virtual_machine",
    "message": ""
  }
}

If valid is false, that resource is noncompliant; if valid is true, it is compliant.

Make changes as needed

Once you’ve tested the custom rule by evaluating the allow, deny, or policy rule, you can make changes to it if you don’t see the expected results. If you launched the REPL with the --watch option, Fregot monitors the loaded files (rules and input) for changes and reloads them automatically.

Changing the input file is handy if you need to test a rule and you don’t have the necessary infrastructure. For example, if you’re writing a rule for the policy “AWS RDS instances should have multi-AZ enabled,” but your environment only has one RDS instance, you can modify the multi_az property from false to true and vice versa in order to make sure the rule works as expected.

You can also copy and paste the resource object starting with the identifier to make a fake one for testing purposes — just make sure to change the resource identifier, and be careful when making edits so the resource is still parsed properly.

You can even watch a specific rule once you’re inside the REPL, and whenever you make changes to any of the loaded files, Fregot reloads them and re-evaluates the rule, printing the updated results. For example, you can use the :watch command as shown below:

:watch allow with input as input.resources["aws_db_instance.xyz123abc456xyz123abc456zyx1234"]

Then you’ll see updated results for the allow rule after you make changes to the Rego file or the input file, as Fregot re-evaluates the rule against the specified resource and prints the new result.

Debug a custom rule

Fregot really shines when you’re troubleshooting a rule. You can enter debugging mode to walk through a function or rule step-by-step, check a property, query, or variable’s value at a given point in the rule, and evaluate queries.

Set and activate a breakpoint

Activating a breakpoint on a program to enter debugging mode is sort of like hitting the pause button on a video. The video runs until you pause it, at which point playback is suspended and you can advance the video one frame at a time, fast-forward to the next scene, or rewind to the previous one. You can also unpause the video and continue playback until the video is paused again or is finished.

Likewise, Fregot evaluates a custom rule using the given input until it gets to the breakpoint, at which point the program is suspended. You can then examine the custom rule one query at a time by stepping forward with the :step command, skip to the next function or Rego rule with :next, return to the previous query or function with :rewind, or continue to the next breakpoint or end of the program with :continue.

To debug a custom rule in Fregot, there are two steps:

1. Use the :break command to set a breakpoint on the function or Rego rule you want to jump into.

2. Activate the breakpoint by entering the name of the function or rule.

Activating a breakpoint puts you into debugging mode, and you can tell because the prompt changes from the name of the package to <name of package>(debug)% as shown below:

vm_size(debug)%

The process for using breakpoints is slightly different for simple and advanced rules.

Set and activate a breakpoint on a simple rule

You can set a breakpoint on any function or Rego rule, but for simple rules, we recommend starting with allow or deny, whichever your custom rule uses.

Step 1: To set the breakpoint on allow or deny, use the :break command with allow or deny:

:break allow

Step 2: To activate the breakpoint, enter the name of the rule you just set the breakpoint on and the resource you want to debug. This tells Fregot to evaluate the given resource and “pause” the evaluation so you can step through it. It’s actually the same command you’d use to test a resource normally, but because we set the breakpoint first, we’ve told Fregot to pause the evaluation so we can walk through it step by step:

<allow or deny> with input as input.resources["<resource_identifier>"]

For example:

allow with input as input.resources["aws_db_instance.xyz123abc456xyz123abc456zyx1234"]

Set and activate a breakpoint on an advanced rule

Again, you can set a breakpoint on any function or Rego rule, but for advanced rules, it’s a good idea to start with policy. Setting a breakpoint on an advanced rule is the same process as for simple rules, but activating the breakpoint differs slightly.

Step 1: To set the breakpoint on policy, use the :break command with policy:

:break policy

Step 2: To activate the breakpoint, simply enter policy. Again, this is just like testing the rule, but because we’ve set the breakpoint first, we’re telling Fregot to pause the evaluation so we can step through it:

policy

Why don’t we need to specify a particular resource to evaluate with an advanced rule? An advanced rule looks at all the resources in the input and generates a policy set containing objects that represent each resource evaluation. When you’re debugging an advanced rule, Fregot steps through the generation of the policy set one object at a time. This means after going through all of the steps to evaluate a resource, Fregot continues to the next resource in the input and steps through the rules to evaluate it. This process continues until Fregot has examined all of the applicable resources. You’re essentially watching as Fregot generates the policy set, containing an object with a compliance decision for each resource.

So, once you’ve “paused” the policy evaluation by activating the breakpoint, you can use :next to make Fregot iterate through all the applicable resources in the input, one at a time. This enables you to not only debug one resource, but any resource of the specified type in the input, all with a single breakpoint.

Listing breakpoints, exiting breakpoints, removing breakpoints

No matter the type of custom rule you’re debugging, you can list all your breakpoints by simply entering :break as shown below:

:break

To remove a breakpoint, just enter :break <function or rule name> again:

:break allow

Note

You can’t reload a rule while you’re in debugging mode, because doing so would modify the code currently running. (Returning to our video metaphor, that would be like trying to record over the video while it’s paused.) Instead, exit debugging mode with the :quit command:

:quit

Then you can manually reload the rule:

:reload

Walk through the rule step-by-step

Within debugging mode, you can use the commands below to “step-through” each part of the debugged rule, following along as Fregot evaluates the input:

• :step proceeds to the next query, rule, or function in the custom rule.

• :next skips to the next complete rule or function in the custom rule.

• :continue continues to the next breakpoint if there is one, or finishes the rule evaluation otherwise.

• :rewind goes back to the most recent :step or :next command.

When you’re debugging a simple rule, Fregot exits debugging mode after stepping through the entire allow or deny rule for the specified resource.

When you’re debugging an advanced rule, Fregot exits debugging mode after stepping through the entirety of the policy set, which includes an evaluation for each applicable resource.

Evaluating properties, queries, and variables in debugging mode

From within debugging mode, you can also evaluate local properties, queries, and variables in context – which means you can look inside a rule/function and find out what a value is at that moment in time, according to the current resource used as input. As with testing, the process is slightly different for simple vs. advanced rules.

Simple rules

Suppose you’ve written a simple rule, aws_rds_backup_retention.rego, which has the policy “AWS RDS instances should have a backup retention period of 14 days or more”:

allow {
  input.backup_retention_period = period
  period >= 14
}

This rule assigns the value of backup_retention_period to the variable period, then checks to see if it’s greater than 14. If so, the resource passes the compliance check; otherwise, it fails.

As stated previously, you can set a breakpoint on any rule or function to access debugging mode. So if your simple custom rule has an allow rule, as this one does, you can set a breakpoint on allow:

:break allow

Then enter allow and a resource identifier to start debugging. For example:

allow with input as input.resources["aws_db_instance.abcDEFghiJKL123456789012mnop123"]

Fregot shows you the first query in the rule, along with the line number it appears on. In this example, the first query of the allow rule is on line 11 and looks like this:

11|   input.backup_retention_period = period
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

From here, you can evaluate a property, query, or variable at that moment in time. For example, we can find out what the value of backup_retention_period is for the resource we paused the evaluation on, or we can even execute arbitrary queries.

For example, if you want to see the value of input.backup_retention_period, you’d use the command below:

input.backup_retention_period

You’d see output like this:

= 7

This shows us that the value of the backup_retention_property property is set to 7 for this particular RDS instance, which we can verify by looking at the input document:

"aws_db_instance.abcDEFghiJKL123456789012mnop123": {
    "_provider": "provider.aws.us-east-1",
    "_type": "aws_db_instance",
    "id": "database-1-noncompliant",
    "backup_retention_period": 7,
<cut for brevity>

In addition to evaluating properties, you can evaluate entire queries. We can check the result of the query on line 11 simply by entering the query:

input.backup_retention_period >= 14

We get this output:

= false

Therefore, the backup_retention_period is not >= 14.

You can evaluate any other arbitrary property or query the same way. Below, we check the value of the engine_version property for the given instance:

input.engine_version

And we see this output:

= "8.0.20"

Likewise, if we wanted to find the result of the query input.engine_version == "8.0.20", we can do it like so:

input.engine_version == "8.0.20"

You’ll see output like this:

= true

We can see that the engine version is indeed 8.0.20.

Note that the property we checked and the query we entered aren’t in the rule. That’s just fine! You can look up anything you like. If the query references properties that don’t actually exist, you’ll just get an empty set to show that there are no matching results:

{}

Variables can be evaluated in the same fashion, but there’s an extra step. If you want to find the value of a variable, you have to step beyond the line where the variable is declared.

Suppose we’d like to see the value of the period variable. As a reminder, this is the line Fregot is examining at the moment:

11|   input.backup_retention_period = period
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Tip

You can use the :where command to show your location in the rule.

When Fregot shows us a line, it’s really showing us what it’s going to do next. That means the line hasn’t actually been executed yet. If we were to try to evaluate the variable now, we’d get an error message:

fregot (compile error):
  "period" (line 1, column 1):
  unknown variable:

    1| period
       ^^^^^^

  Undefined variable: period

Because Fregot is showing us the line where we assign the period variable, we need Fregot to move to the next step of the evaluation so the assignment gets carried out. We use the :step command to move forward:

:step

We see this output:

12|   period >= 14
      ^^^^^^^^^^^^

The variable has officially been declared and assigned a value, so by entering the variable name, we can look up its value:

period

We see this output:

= 7
| period = 7

Note that when you evaluate a variable, a property of a variable, or a query involving a variable, Fregot gives you the info you’ve requested and then prints the entire variable’s value underneath. This is useful when you’re evaluating a query or just the property of a variable, because it gives you meaningful context – handy when a variable represents a resource, as it does in advanced rules (and which we’ll demonstrate next). In cases where you’ve requested the entire value of the variable, as above, you’ll just see the information repeated.

Advanced rules

A policy rule typically includes a line like this:

single_resource = resource_collection[_]

The main thing to remember when debugging an advanced custom rule is that you need to proceed past this line in order to look at an individual resource. Once Fregot has isolated a resource, you can evaluate a property or variable simply by entering its name, or even evaluate an entire query, as you would for a simple rule. You can proceed past this line by using the :next command:

:next

And you’ll see the next line of the policy rule. For example:

14|   vm.delete_os_disk_on_termination == false
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Why not :step? The :step command executes every single line in every single rule or function, including ones referenced in libraries. In contrast, the :next command skips to the next complete rule or function in the program, and this is desirable here because :step actually enters the fugue.resources(resource_type) function in the fugue.rego library to do the business of collecting the resources. You can tell because the command prompt looks like this:

fugue(debug)%

Instead of this:

<your package name>(debug)%

If you happen to forget and execute :step, that’s no problem; just enter :next until you return to your rule’s package.

When Fregot has finished evaluating an individual resource, it returns to the single_resource = resource_collection[_] line to iterate through the collection and single out a new resource.

That’s why the other key thing to remember when debugging an advanced rule is that when you proceed past the single_resource = resource_collection[_] line again, that signals a new resource is being evaluated, and when you enter a property name, for example, you’ll see the results for the new resource.

The upshot is that you can enter :next over and over to walk through each resource evaluation, watching as Fregot generates the policy set object by object.

To demonstrate how to evaluate local expressions when debugging advanced rules, let’s take a look at the logic for an advanced rule, azure_vm_delete_on_terminate.rego. This rule has the policy “Azure virtual machines should not delete the OS disk upon termination”:

vms = fugue.resources("azurerm_virtual_machine")

policy[r] {
  vm = vms[_]
  vm.delete_os_disk_on_termination == false
  r = fugue.allow_resource(vm)
} {
  vm = vms[_]
  not vm.delete_os_disk_on_termination == false
  r = fugue.deny_resource(vm)
}

It looks at all the virtual machines in the environment and checks whether delete_os_disk_on_termination is set to false, and if so, the resource passes. Otherwise, it fails.

Let’s set a breakpoint on policy:

:break policy

Then we’ll activate the breakpoint:

policy

Fregot displays the following line, which is where we isolate a virtual machine from the collection to evaluate it:

13|   vm = vms[_]
      ^^^^^^^^^^^

Enter :next to proceed:

:next

You’ll see this output:

14|   vm.delete_os_disk_on_termination == false
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Because you’ve proceeded past the vm = vms[_] query, Fregot is looking at a specific resource. This means you can enter vm.delete_os_disk_on_termination to see what the value is for the currently evaluated resource in the input:

vm.delete_os_disk_on_termination

(This is just like how you’d check a property when debugging a simple rule, but since it’s an advanced rule, the single_resource variable is used instead of input.)

You’ll see this output:

= false
| vm = {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
    "storage_image_reference": [{
      "id": "",
      "publisher": "Canonical",
      "offer": "UbuntuServer",
      "sku": "18.04-LTS",
      "version": "latest"
    }],
<--cut for brevity-->

That’s a lot of output! When you enter a variable, a query involving a variable, or just a property of a variable (as we’ve done here), Fregot prints the information we’ve asked for followed by the entirety of the variable for context. This is like when we evaluated the period variable in the simple rule, but this time, the vm variable represents an entire resource, which we can see above.

Because the vm variable represents an entire resource, you can enter the name of the variable to see its entire value – aka, all of its properties:

vm

You’ll see more output:

= {
  "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
  "storage_image_reference": [{
    "id": "",
    "publisher": "Canonical",
    "offer": "UbuntuServer",
    "sku": "18.04-LTS",
    "version": "latest"
<--cut for brevity-->
| vm = {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
    "storage_image_reference": [{
      "id": "",
      "publisher": "Canonical",
      "offer": "UbuntuServer",
      "sku": "18.04-LTS",
      "version": "latest"
    }],
<--cut for brevity-->

Just like with simple rules, you have to proceed past the line where the variable is declared in order to evaluate any expression involving it. We can check the value of the vm variable because we’ve proceeded past the vm = vms[_] line.

If at any time you’re not sure which resource Fregot is currently evaluating, you can enter single_resource_variable.id like so:

vm.id

Fregot displays the resource ID followed by the entirety of the variable again:

= "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm"
| vm = {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
    "storage_image_reference": [{
      "id": "",
      "publisher": "Canonical",
      "offer": "UbuntuServer",
      "sku": "18.04-LTS",
      "version": "latest"
    }],
    <--cut for brevity-->

If the resource shown isn’t the one you want to check, simply advance through the queries with :next until Fregot starts over at vm = vms[_], which means it’s evaluating a different resource.

As with simple rules, you can evaluate entire queries, and they don’t have to be in the custom rule – they can be arbitrary:

vm.vm_size == "Standard_B1ls"

We can see the result of this query is true, and we see the entire variable for context again:

= true
| vm = {
    "id": "/subscriptions/12345678-1234-5678-9012-123456789012/resourceGroups/example-rg/providers/Microsoft.Compute/virtualMachines/compliantvm",
    "storage_image_reference": [{
      "id": "",
      "publisher": "Canonical",
      "offer": "UbuntuServer",
      "sku": "18.04-LTS",
      "version": "latest"
    }],
    <--cut for brevity-->

This process of debugging a custom rule gives you visibility into how a resource’s compliance state is calculated and allows you to more effectively catch bugs, whether simple or advanced.

Using debugging mode to find the right syntax for a property

One useful tip is that in debugging mode, you can examine any property in the currently evaluated resource. For example, if you wanted to see what the value of the “application” tag is for a VM in a simple rule, you’d enter the following:

input.tags.application with input as input.resources["azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p"]

And you’d see output like this:

= "portal"

This is especially useful when you’re not sure how to get the syntax of a property just right. For example, you can return the entire input.tags object for the VM like so:

input.tags with input as input.resources["azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p"]

You’d see:

= {
  "application": "portal",
  "Name": "VM"
}

You can then try different things to produce the results you’re looking for. For example, you can iterate through each tag value:

input.tags[_] with input as input.resources["azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p"]

You’d see:

= "portal"
= "VM"

Or, you can return the value for just the tag key Name:

input.tags.Name with input as input.resources["azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p"]

= "VM"

If you forget the syntax and try something like this:

input.tags.Name[_] with input as input.resources["azurerm_virtual_machine.A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p"]

You’ll see an empty set:

{}

What’s next?

For more information about Fregot, see the README.