And So It Came To Be

Some time ago Denis Budyak posted on the Iterate devel mailing list regarding a patch that would allow Iterate to use keyword symbols for drivers and clauses. The Iterate devel mailing list was in general, and the current maintainer in particular, very quiet regarding this matter, scarily so, even. But I and one other user spoke out that we thought this was a bad idea. This is not exactly a consensus, but here were the basic arguments. Denis wanted to patch Iterate such that this was valid syntax.

(iterate
  (:for (x y) :in seq)
  (:while (some-predicate-p y))
  (:collect (list (f x) (g y))))

This removes one of the common annoyances that new users of Iterate have to grapple with; the iterate package exports a lot of symbols with very common names. For instance, people often times implement their own while macro as part of an introductory text. Iterate has its own while loop macro and this leads people to complain that they cannot have their homebrew:while and iterate:while imported into the same package (why they don't complain about other symbol names conflicts is a bit lost on me, I remember writing my own with-gensyms that I was relatively proud of). This patch would be a way around this.

My argument for not doing this was primarily that it would bifurcate the method by which Iterate is extended. For those that are not aware, Iterate is a DSL like Loop, but it is not as disjoint from standard Lisp as Loop is. Iterate is fairly transparently built out of standard Lisp macros and functions and, as such, Iterate is extended by writing standard Lisp macros and functions. Because of this "Lispy" design, symbols are not just syntactic markers for the DSL, they actually have meaning in the Lisp system at large. In this framework, the package system gives us the benefits of a divided name-space (as much benefit as it can give until the nickname business is figured out).

But in the end, I had to admit that this change wasn't likely to actually break anything so long as the original syntax was still allowed. I still oppose this change on the basis that it would muddy the waters of extension, raising the predefined drivers and aggregation clauses like for, while, and collect to a higher, irreplaceable status. For example, this would mean that I couldn't redefine the for driver. Even with all of this in consideration, I had to admit, this was a weak argument and these problems would never actually come up. Turns out, I was wrong.

Let's say, for instance, we find something that is more or less a bug in iterate:for. Let's say we find something that is basically just not the way that it should be done. For instance, why doesn't for accept normal destructuring-bind style argument lists?

The destructuring-bind construct will take apart a lambda list like the ones used in the defmacro forms. The destructuring-bind construct is a basic part of Lisp. And yet this will produce an error in standard Iterate:

(iterate
  (for (x &rest y) in-sequence #((1 2) (3)))
  (for (&key (a 3) b) in '((:a 1 :b 2) (:b 3)))
  (collect (list x y a b)))

That just seems like it should work. Not only does it seem like it should work, it would be pretty darn useful if it did work. For instance, it would be easier to use lists as general purpose data containers because we can use keyword meta-data without an extra destructuring-bind cluttering the code. We can also use optional bindings, something that shows up a lot in the destructuring I do. This method of destructuring is inherently more powerful than the facility that Iterate currently has and it's more consistent with standard Lisp syntax.

This brings me to the point. Iterate has a defined built in behavior which I don't agree with. To me this feels like a bug, something that should be fixed. However, this fix is not compatible with the current behavior. One of the interesting aspects of Common Lisp is that any enhancement at all is usually not backward compatible because errors have a defined run time consequence. This means that there is a pretty good argument to not "fix" this. This seems at first like an impasse, and it would be if we were using Loop or a version of Iterate with the proposed patch. The extensibility of Iterate can save us here.

To summarize my options, I could:

1. Patch my local Iterate so this works. This means that my code will not work with anybody else's version of Iterate. This is a non-option really.
2. Write a patch and petition the Iterate developers and community at large to accept this change. This is more attractive, but still deeply flawed. First, because errors have defined run-time behavior in Common Lisp, this is technically an incompatible change. Second, seeing as the maintainer didn't even weigh in on the previous discussion, I don't hold high hopes that any patch would be accepted in a timely manner.
3. Or, simply replace the for driver with my own.

This last option means I am able to make the change I want without approval from a community or maintainer (which may or may not be absent) and am able to do so in a completely portable, completely backward/forward/concurrently compatible way. With this method, I can make the change and start using it immediately in my own code. Meanwhile, I can publish the change and the maintainer can contemplate the benefits of such a change, and the community can make the decision for their selves on an individual by individual basis. In order to use it, just (shadowing-import smithzv.destructuring-bind-iterate::for), or when you define your package:

(defpackage my-package
  (:use :cl :iterate)
  (:shadowing-import-from :smithzv.destructuring-bind-iterate #:for))

If you want both behaviors, simply use smithzv.destructuring-bind-iterate which will import the new macro under the name dfor. Further, if the maintainer ever wanted to include this in Iterate, they have a working implementation that can be easily adapted to their code.

I am able to do all this because of the framework of extension that the proposed patch would uproot, or at the very least weaken. With the built-in macros and functions in Iterate implemented as keyword symbols, it would be impossible to create a drop-in replacement like this. I guess that in the end, this is the best argument I have as to why this change shouldn't be made.