jfmengels' blog

2.3.0 - Just try it out

September 08, 2020

This is going to be another simultaneous release for both the elm-review CLI and the jfmengels/elm-review Elm package. Both bring what I think are very exciting features. Let’s start by going through what happens in the CLI.

elm-review CLI

Using elm-serialize

The CLI got a few new performance improvements. The most notable one was made possible with Martin Stewart’s help by integrating his own elm-serialize library.

Under the hood, elm-review parses the files in your project using elm-syntax, and then stores those as JSON deep inside your elm-stuff folder. During future runs, these will be read from the file system so that we don’t have to do the parsing again which is quite slow. This is one of the reasons that elm-review is faster after the initial run.

Using elm-syntax’s default encode/decoder functions, the stringified AST is considerably larger than the code it represents. For the codebase I work on daily (~170k LOC), it translates to a quite large 40MB of disk space, and writing/reading that amount of data from the file system takes a non-negligible amount of time.

elm-serialize produces a significantly compressed JSON output (or binary, but which proved to be less efficient at this time). Instead of {"expression": {"type": "Operator", "operator" : "+" } } it will be something like [0,[0,0,1,"+"]].

Overall, this reduces the total cache size from 40MB to 14MB in the large project I mentioned, and because there is less data to read from (and to write to) the slow file system, the total run time was reduced by about 15%!

A side-note on elm-optimize-level-2

I also tried integrating elm-optimize-level-2, which seems to improve the speed of the review phase by about 10%. It unfortunately takes 1.5 to 2 seconds to optimize the compiled configuration, which I estimate is a bit too slow for something that may regularly be compiled. But I will keep an eye on it, as I imagine more and more optimizations will be applied and the optimization time will be reduced.

--rules

The first addition to the CLI capabilities is the --rules flag. This allows you to run only the chosen rules from your review configuration.

elm-review --rules New.Rule1,New.Rule2

This can be useful for instance to reduce the amount of noise when you want to introduce elm-review (or several new rules) to a codebase by concentrating on fixing the reported errors one rule at a time.

--template

Alright, this is the big feature of this CLI release. The newly added --template flag allows you to try out elm-review without any prior setup.

With --template, you can use elm-review with a remote configuration you found on GitHub. For instance, if I had stored (I don’t though) the review configuration I use in most of my projects on GitHub in the jfmengels/elm-review-configuration repo, I could review one of my projects with this configuration without having previously configured elm-review by running the following command:

elm-review --template jfmengels/elm-review-configuration

You can specify a sub-directory inside that repository. Review packages will usually contain an example/ folder which provides an example configuration (the same as the one you’d see in the README). You can use this to try out the rules inside that package without adding the package as a dependency and adding the rules to your review configuration, or even without setting up anything in your project.

elm-review --template jfmengels/elm-review-unused/example

If needed you can also specify a branch, tag or commit.

elm-review --template jfmengels/elm-review-unused/example#new-rule-in-progress
elm-review --template jfmengels/elm-review-unused/example#1.0.0
elm-review --template jfmengels/elm-review-unused/example#7ae7a8

This is really useful if you want to try out a review package and see if and what it would report in your project. You can combine this with the previously introduced --rules to only run a single rule, which is what you’ll see in the documentation of the rules:

elm-review --template jfmengels/elm-review-unused/example --rules NoUnused.Variables

Lastly, --template also works with elm-review init, which basically lets you copy-paste a remote configuration inside your project.

elm-review init --template jfmengels/elm-review-unused/example

Ok, let’s summarize this feature!

# Run elm-review
elm-review
# Haven't installed elm-review yet? Use npx
npx elm-review   # (does require Node.js)
# Haven't set up a configuration either?
npx elm-review --template jfmengels/elm-review-unused/example
# Oh you just want to try out that one rule?
npx elm-review --template jfmengels/elm-review-unused/example --rules NoUnused.Variables
# You like that configuration? Configure your project using it
npx elm-review init --template jfmengels/elm-review-unused/example

Note that if you run this too often, GitHub will block you because of rate limiting (for about 30 minutes), so prefer using a local configuration as a more permanent solution 😉

I’ll talk about it in the next section, but you can also use --template to run yet unpublished rules!

new-package

In version 2.2.0 I introduced new-package 📦. With this version, it will help create an even more complete experience, especially with regards to providing the example configuration that users will need to try out the package using --template. All in a way that will be useful to the maintainer instead of being a chore 🧙.

new-package now creates a few additional files: It creates a maintaining document which explains how to get started, how to publish, maintain the example configurations, etc. new-package also creates a few scripts that will help with the maintenance of said example configurations. Lastly, it will create a preview/ configuration.

new-package will now also create a preview/ configuration, similar to the example/ configuration. The difference between the two is the following: example/ works with the latest released version (it includes the rules by depending on the review package), whereas preview/ works with the latest source code (it includes the rules by including them in the source-directories).

The goal for example/ is to reflect what users would have by adding the review package configuration, whereas the goal for preview/ is to allow people to try out rules at any time, even before they get released initially.

If you open an issue for a rule, the maintainer can then fix the problem and ask you to try out the rule and validate that the fix solved your issue before a new version of the package gets released. Reviewing using an unpublished rule works just like running an example rule using --template, except that you replace “example” by “preview”:

elm-review --template <author>/<package name>/preview --rules <rule name>

I imagine that maintaining two sets of configurations would be tedious for maintainers, so new-package provides scripts to make it really easy to create the example configuration from the preview configuration, and the test setup makes sure that you have done everything needed when it is needed. I explain this in more details in this section of the maintenance guide.

I recommend maintainers of already published review packages to run new-package anew, see what changed and take what they think is appropriate. It would be very helpful for the users if you could add the “Try it out” sections in the README and in each rule’s documentation. Ask for help in the #elm-review Slack channel if needed!

Notes on the CLI release

That was it for the notable and visible parts of the CLI. Note that starting from 2.3.0, the CLI will only work if your review configuration depends on jfmengels/elm-review 2.3.0 and later, which I am releasing today too and which we will delve into now.

elm-review package

In this jfmengels/elm-review package, I added the concept of context creators. In previous versions, when you would create a rule (technically, a module rule), you would need to specify the initial context in the form of a static value.

rule : Rule
rule =
    Rule.newModuleRuleSchema "My.Rule.Name" initialContext
        -- |> ... visitors
        |> Rule.fromModuleRuleSchema

initialContext : Context
initialContext =
    { currentModuleName : [] }

Context creators grant the ability to initialize the rule using a function, where you tell what pieces of information you’d like to be provided in a JSON-decode-pipeline-like API.

rule : Rule
rule =
    Rule.newModuleRuleSchemaUsingContextCreator "Rule.Name" contextCreator
        -- |> ... visitors
        |> Rule.fromModuleRuleSchema

contextCreator : Rule.ContextCreator () Context
contextCreator =
    Rule.initContextCreator
        (\metadata moduleNameLookupTable () ->
            { currentModuleName = Rule.moduleNameFromMetadata metadata
            , moduleNameLookupTable = moduleNameLookupTable
            }
        )
        |> Rule.withMetadata
        |> Rule.withModuleNameLookupTable

So what data can you get access to?

Through withMetadata, you can ask for the name of the module (or for the node of the module name), for which you previously had to add a module definition visitor and set an awkard default value in the initial context. You can also ask for whether the module is part of the project’s source directories (src/ for packages) or not. This can be useful if you want different behaviors for test files.

The most exciting one for me, is getting the module name lookup table through withModuleNameLookupTable. If you have used elm-review-scope, this solves the same issue but in a now native and more exact way, and you should be able to switch quite easily to using the lookup table. The issue that that solves is knowing what module a type or value comes from when you see a reference to it. For instance, if in the code you see A.b or b, the lookup table will resolve the name of the module where b has been defined, based on the declarations, imports and aliases in the project. It’s a surprisingly tricky problem that you don’t want to handle on your own 😅

Here is an example of how that would be used:

type alias Context =
    { lookupTable : ModuleNameLookupTable }

contextCreator : Rule.ContextCreator () Context
contextCreator =
    Rule.initContextCreator
        (\lookupTable () -> { lookupTable = lookupTable })
        |> Rule.withModuleNameLookupTable


expressionVisitor : Node Expression -> Context -> ( List (Error {}), Context )
expressionVisitor node context =
    case Node.value node of
        Expression.FunctionOrValue _ "color" ->
            if ModuleNameLookupTable.moduleNameFor context.lookupTable node == Just [ "Css" ] then
                ( [ Rule.error
                        { message = "Do not use `Css.color` directly, use the Colors module instead"
                        , details = [ "..." ]
                        }
                        (Node.range node)
                  ]
                , context
                )

            else
                ( [], context )

        _ ->
            ( [], context )

Instead of running the visitors that collect the information to resolve the module name in every rule like what was done with elm-review-scope, this data will be collected once and then provided to all the rules that demand it. Through the use of context creators, elm-review can even skip the computation if no rule in the configuration asks for it.

Context creators are also available for project rules when you do withModuleContextUsingContextCreator in the fromProjectToModule and the fromModuleToProject functions. In these functions, you also have access to the project context (or module context for fromModuleToProject). You also have access to the module key just like in withModuleContext. Using a context creator allows you to simplify those functions by not having to ignore the arguments you don’t care about.

I hope to later allow for more information to be made easily available (based on how many use-cases that require it), such as easily knowing what is exposed in a module or even what the type of an expression is, likely through a new kind of lookup table and accompanying functions.

Truth be told, I am not a fan of the names of most of these functions, especially the really long ones. I think that I will introduce aliases as soon as I find more fitting names and remove the current names in the next breaking change. Feedback welcome!

elm-review-unused

While I’m at it, jfmengels/elm-review-unused got a new rule: NoUnused.CustomTypeConstructorArgs.

This will help discover unused arguments for custom type constructors.

type CustomType
  = CustomType Used Unused -- Unused will be reported

case customType of
  CustomType value _ -> value

As for a lot of the rules in elm-review-unused, each rule doesn’t discover much on its own, but helps uncover new things that the other rules can then detect, and then you have a snowball effect.

I asked people to run this code on their project, and Simon Lydell reported back his results:

44 errors in 20 files. 60k sloc. Awesome results:

The rule found lots of unnecessary stuff! Removing it allowed simplifying some decoders greatly, and even getting rid of an unnecessary dependency and some then-unused functions!

The rule found one place where we accidentally rendered Element.none instead of an error message.

The rule found two places where we forgot to send errors to Sentry.

It found one potential bug where we might show the wrong string due to a Foo \_ -> pattern.

Help out

I hope that these features are as exciting for you as they are for me. If you didn’t before, I hope you will try elm-review out, especially since it has now become really easy to try it out.

If you wish to help out, join the #elm-review channel on either the Elm Slack or the Incremental Elm Discord, and let me know. I plan on making it easy to contribute and find tasks for Hacktoberfest. Please consider supporting me financially if you or your company benefit from the tool, as that also helps a lot.

elm-review will continue to evolve in very interesting and novel ways. Come take part of the journey 😊

Written by Jeroen Engels, author of elm-review. If you like what you read or what I made, you can follow me on BlueSky/Mastodon or sponsor me so that I can one day do more of this full-time ❤️.