Some Emacs hacks for GDB and other stuff

I am fighting with GDB in Emacs today. Thought I would share a few hacks that make things more livable. These get hackier as you go, so… be warned.

First, GDB in many windows mode makes a bunch of dedicated windows. This can be pretty annoying. In fact, dedicated windows in general can be pretty annoying. If I want to change the buffer of a window, don't you dare stop me. Emacs Wiki has some advice code that allows you to disable the setting of dedicated windows in GDB startup (bottom of the page). But to be clear, this is a problem with the Emacs interface, not the GDB interface. The GDB interface has a good reason to not want those buffers to change, and it is annoying to use it if they do change. The problem is when I know that I really do want these to change but Emacs makes me jump through hoops to do it. So, let's fix the underlying problem. Here is a bit of code to add to your init files that will allow you to change buffers even if they are dedicated.

(defun undedicate-window (&optional window)
  (interactive)
  (set-window-dedicated-p (or window (get-buffer-window)) nil))

;; Removing annoying dedicated buffer nonsense
(defun switch-to-buffer! (buffer-or-name &optional norecord force-same-window)
  "Like switch-to-buffer but works for dedicated buffers \(though
it will ask first)."
  (interactive
   (list (read-buffer-to-switch "Switch to buffer: ") nil 'force-same-window))
  (when (and (window-dedicated-p (get-buffer-window))
             (yes-or-no-p "This window is dedicated, undedicate it? "))
    (undedicate-window))
  (switch-to-buffer buffer-or-name norecord force-same-window))

I just set a global key binding of (kbd "C-x b") to switch-to-buffer! (actually I use a global minor mode to keep track of all of my keybindings, but the result is the same). This will now act exactly like switch-to-buffer unless the window is dedicated, in which case it will ask if you want to "undedicate" the window first. Now Emacs wont reuse these buffers willy nilly, but you can still do what you think is best.

Second, dedicated windows are so convenient (now that they are convenient to undedicate) that you might find that you want to start setting the dedicated flag on random windows that you don't want Emacs to change. So here is a function for that. If you want a dedicated completion window, well, just set it on that window and you won't have to worry about it getting taken over by some other Emacs pop-up.

(defun toggle-window-dedication (&optional window)
  (interactive)
  (let ((window (or window (get-buffer-window))))
    (set-window-dedicated-p window (not (window-dedicated-p window)))))

(global-set-key (kbd "C-x d") 'toggle-window-dedication)

Third, I really hate that performing any act in GUD tends to make the source buffer you are working with jump back to the execution point. This is a problem if you are setting several breakpoints, or printing out several values from the source file. Thus I came up with this hack (and this really is a hack) to make this problem go away.

(defun nice-gud-print (arg)
  (interactive "p")
  (save-excursion
   (gud-print arg)
   (sleep-for .1)))
(global-set-key (kbd "C-x C-a C-p") 'nice-gud-print)
(defun nice-gud-break (arg)
  (interactive "p")
  (save-excursion
   (gud-break arg)
   (sleep-for .1)))
(global-set-key (kbd "C-x C-a C-b") 'nice-gud-break)
(defun nice-gud-tbreak (arg)
  (interactive "p")
  (save-excursion
   (gud-tbreak arg)
   (sleep-for .1)))
(global-set-key (kbd "C-x C-a C-t") 'nice-gud-tbreak)

The sleep-for call is necessary to make save-excursion actually work. My guess here is that GUD queues requests and then executes them later. So without the sleep, my call to GUD returns, and with it the save-excursion environment exits, long before GUD processes the command and then resets the view to the execution point. Not pretty, but it works for me at least.

Lastly, I find that it is helpful to have a key binding for gdb-restore-windows. This way you can easily jump to a particular window, make the buffer full screen, and then return to the debugger view later. However, if you like to use a vertically split screen like I do (i.e. you like 80 column width programs), it is often even better to have a toggle between just the two center windows and the full debugger view. So, here is a function to do that:

(defun gdb-windows-toggle (&optional full-restore)
  (interactive "P")
  (let ((window-tree (first (window-tree))))
    (cond (full-restore
           (gdb-restore-windows))
          ((and (listp window-tree)
                (= (length window-tree) 5))
           (let ((buffers (if (listp (fourth window-tree))
                              (mapcar 'window-buffer
                                      (rest (rest (fourth window-tree))))
                              (list (window-buffer (fourth window-tree)))))
                 (first-selected (or (windowp (fourth window-tree))
                                     (eql (first (window-list))
                                          (third (fourth window-tree))))))
             (delete-other-windows)
             (when (> (length buffers) 1)
               (split-window-horizontally))
             (cl-loop for buffer in buffers
                      for window in (window-list)
                      do (progn (undedicate-window window)
                                (set-window-buffer window buffer)))
             (unless first-selected
               (select-window (second (window-list))))))
          ((or (windowp window-tree)
               (and (= (length window-tree) 4)
                    ;; horizontal split
                    (not (first window-tree))
                    ;; No further splits
                    (windowp (third window-tree))
                    (windowp (fourth window-tree))))
           (let ((current-buffers
                   (if (windowp window-tree)
                       (list (window-buffer window-tree))
                       (mapcar 'window-buffer (rest (rest window-tree)))))
                 (first-selected (or (windowp window-tree)
                                     (eql (first (window-list))
                                          (third window-tree)))))
             (gdb-restore-windows)
             (let ((windows (rest (rest (fourth (first (window-tree)))))))
               (when (= (length current-buffers) 1)
                 (delete-window (second windows)))
               (cl-loop for buffer in current-buffers
                        for window in windows
                        do (progn (undedicate-window window)
                                  (set-window-buffer window buffer)))
               (if first-selected
                   (select-window (first windows))
                   (select-window (second windows))))))
          (t ;; If all else fails, just restore the windows
           (gdb-restore-windows)))))

(global-set-key (kbd "C-c g") 'gdb-windows-toggle)

That is long and ugly, and probably could be made much simpler, but it does the trick and does it pretty well. Maybe if these really work out I can get someone involved with Emacs to include some of these little hacks (cleaned up of course). It would be nice to deal with the fact that GDB is fundamentally broken for me right now, but… we'll see where those bug reports go.

Rudel Survival Guide

We are gearing up for our collaborative effort for ICFP and so I figured it might be nice to write out a little "how to make it work" guide for Rudel, the collaborative editing framework for Emacs. To get this out of the way first, Rudel doesn't work out of the box. In order to use it, we had to hack a bit on the source to correct what we can only guess are legitimate errors. That said, I certainly don't have the expertise on the system in order to dictate the correct way to solve these problem.

As a note, throughout this contest our setup will be:

• Rudel v0.3
• Obby backend
• TCP transport
• Port 6522
• not using the built in Rudel/Obby encryption
• all connections must be tunneled over SSH (for encryption/access control)
• No global or user passwords

The first step for using Rudel is to, naturally, install Rudel. If you have Emacs v24, you may use the package manager via M-x package-list-packages. This will make sure that it also gets any dependencies, however I don't think there are any that aren't already part of Emacs. If you are not using Emacs v24, you will need to install package.el (and this will actually be useful as I believe it will have to track down some dependencies). This can be done like this:

cd .emacs.d
wget http://repo.or.cz/w/emacs.git/blob_plain/1a0a666f941c99882093d7bd08ced15033bc3f0c:/lisp/emacs-lisp/package.el

Then from Emacs, M-x load-file that package.el file. Then you can use continue as if you were using v24 (except where noted).

Rudel is available via the the Marmalade repository. In order to enable the Marmalade repository, you should add something like this early in your .emacs file:

;; (load "~/.emacs.d/package.el") ;; If not v24

;; Load up the new package manager
(require 'package)
;; Add the Marmalade repo
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/") t)
(package-initialize)

The install/compile should seemingly complete without any issues. (If you are not on v24, then there might be an issue with package displaying the info page of certain packages, Rudel amongst them. Instead of using M-x package-list-packages, use M-x package-install and specify "rudel" when it asks).

Now the trouble fun begins.

1.1 We broke your Emacs

Try closing and restarting Emacs. On your next start, Emacs will fail to read your .emacs file with an error like:

Symbol's function definition is void: eieio-defclass-autoload

This is because there are some bugs in Rudel that make it not load properly. My solution is to not use the standard method of loading Rudel. The first order of business is stopping it from loading the correct but broken way via the package manager. Go into the shell and move the rudel directory somewhere where it cannot be found by the package manager:

cd .emacs.d
mv elpa/rudel-0.3 ./

Now Emacs should read your .emacs without issue (because it no longer is trying to load Rudel).

The next order of business, we want to be able to load and use Rudel. In order to do this, we will run Rudel's compile script. Do a M-x load-file on .emacs.d/rudel-0.3/rudel-compile.el. This will compile and generate some files, most importantly for our purposes, it will generate rudel-loaddefs.el. Perform a M-x load-file on .emacs.d/rudel-0.3/rudel-loaddefs.el and Rudel should be operational. Try it out. Use M-x rudel-host-session to start a session (protocol obby, transport tcp, port 6522). Then join that session via M-x rudel-join-session. Try publishing a buffer with M-x rudel-publish-buffer. This should all work.

We want to make this permanent, so we should add something like:

(load-file "~/.emacs.d/rudel-0.3/rudel-loaddefs.el")

…to our .emacs file after the (package-initialize) statement.

Look, I know this is extremely hackish, but I think it will work for everybody. It is the only way I have consistently been able to get things working.

1.2 Joining and leaving sessions

So, as best I can see, this is the status of this functionality in Rudel: you can join sessions and you can leave, but as far as the server cares, you can never leave. This doesn't seem like too much of a problem at first, but here is how problems start to manifest.

1. A username must be unique: This means that each time you log in, you have to pick a unique name, not the one you used last time. This manifests as a lot of "smithzv", "smithzv2", "smithzv3", etc. Luckily, you shouldn't be disconnected often.
2. A color must be unique at login time. This one isn't as bad as you can change colors once you are in the session using rudel-change-color. This means that a good practice is to log in and pick a garish and otherwise infrequently used color and then immediately change it to something more appropriate. No need to worry about conflicts after you have logged in.

1.3 Undo and Rudel

So, one of the biggest problems that I have with collaborative editing in Rudel is that undo are treated very literally. I you undo, you implicitly expect it to apply to your code. However, with Rudel, where someone could be editing somewhere else in the file, undo is happy to go about undoing their work as they type rather than the edits you just made.

The end result is that in a collaborative editing environment, undo is a very dangerous thing; doubly so when undo means what it does in Emacs land (i.e. you can redo by undoing an undo). Basically, if you are using Rudel, you probably should not be using undo, at all. This is a pretty tall order if you have deeply internalized undo into your editing commands (as I have).

In strikes me that a helpful heuristic would be to undo only within a region that contains only your edits (or even using something like multiple-regions to allow for undo of anything that you have edited but not things that others have). This means that if you are the only person editing a buffer, undo works exactly as expected, but if there are others editing the buffer, it will only undo your edits. Note that this isn't always what you want.

However, I'm not sure that such a heuristic is possible (or more likely, it is possible but is hard to do). I'll take a look. It seems that for safe, useful undoing, you need to tell everybody that is working on that buffer that they need to stop what they are doing so you may perform your undos.

I realize that other undo systems can be used with Emacs. For instance, there is Undo-Tree. I am not sure how well these work (or really how they work). Perhaps someone who is better versed in these tools can enlighten us.

1.4 When things go wrong

There are times when, try as it might, Rudel screws up and the files become out of sync. This happens fairly infrequently, thank goodness, but when it does, Rudel does not handle this gracefully. There is no "resync" function that I can see. This means that if you find that your buffer says one thing, but someone elses says something else (this usually manifests a what looks like a typo, but if you fix it, another person goes back and changes it back to its typo state), you will have do something pretty drastic in order to get Rudel to work correctly again. There are a couple of things that must work, but they both are pretty annoying:

1. Ditch this buffer, have everybody unsubscribe, rename the buffer so it will share under a different name, then republish. This way everything has started fresh with this buffer.
2. Restart the entire Rudel session.

Of course, the first method is preferred to the second.

1.5 Dealing with Annoying Colors

If you start using Rudel, sometimes a collaborator will pick a color that doesn't work with your theme or general aesthetic. Luckily, there is something you can do about it even if they refuse to change their color (or are perhaps away from keyboard), you can turn off all coloring by author. Simply specialize the variable rudel-overlay-author-display (set it to nil) and no more colors. This is pretty necessary right now because Rudel is ignorant of the difference between light and dark themes. Thus two locally appropriate choices might be completely inappropriate for the remote party.

Some Customizations to W3m-Mode

For many tasks, I find that it is easier to use a text mode browser in Emacs than it is to use an external browser. This is clearly the case when you would like to copy some text from a web page into an Emacs buffer, like when you are grabbing code examples from a manual or tutorial. One of the main uses I have of this is to quickly browse the Common Lisp Hyperspec. I have found that W3m and W3m-Mode is quite good for these tasks. W3m is a text mode browser that exists independently from Emacs. W3m-Mode is a mode that connects to W3m, sends requests from Emacs to the external process, and receives the output and displays it in an Emacs buffer. Edit: If you want to set Emacs up to use this browser, or any browser, customize the browse-url-browser-function variable.

Of course W3m won't work with a lot of "modern" websites, anything that uses Flash (of course), or anything that utilizes Javascript extensively. Even with all of those barriers, it works surprisingly well because much of the information that is out there on the Internet (and most of the information that I consume using this browser) is actually just text. Further, thanks to HTML5 and the return of the separation of content (HTML) and design (CSS) many websites can be rendered usefully in text.

However, as much as I have learned to like W3m-Mode, there are some annoying aspects of the mode that are the default. Namely, I would like to be able to move through the buffer with the standard directional keys, as well as switch tabs in the way that most other browsers do via "C-<tab>", and move forward and backward in history via "M-<right>" and "M-<left>", respectively. With a little help from the Emacs folks on /r/emacs, here are some bindings that will set this up for you.

;; W3m for browsing
(defun w3m-mode-options ()
  (local-set-key (kbd "<left>") 'backward-char)
  (local-set-key (kbd "<right>") 'forward-char)
  (local-set-key (kbd "<up>") 'previous-line)
  (local-set-key (kbd "<down>") 'next-line)
  (local-set-key (kbd "M-<left>") 'w3m-view-previous-page)
  (local-set-key (kbd "M-<right>") 'w3m-view-next-page)
  (local-set-key (kbd "C-<tab>") 'w3m-next-buffer)
  (local-set-key (kbd "q") 'bury-buffer))
(add-hook 'w3m-mode-hook 'w3m-mode-options)

That last local-set-key is to rebind the "q" key. By default, "q" tells W3m-Mode to kill all W3m sessions and close the window. This isn't the right thing to do for a couple of reasons. First, if I want to kill all W3m sessions, I will do so by killing the buffer (which will kill the sessions). I don't need a separate key for that. Second, closing windows without the user expressly asking for a window to be closed is not good Emacs interface style. The correct thing to do is to go away (whatever that means) and leave the window to be filled with another buffer. The only exception to the rule is popup windows, windows that are meant to be extremely short lived (less than a minute) and which created the window in which the reside. W3m is most definitely not a popup window. Looking at this, I felt that the best solution is to map "q" to bury-buffer and leave it at that.

There are a few other bindings that are different from the standard key bindings that a browser uses, including making a new tab and kill that tab. These exist and are reasonable (they fit in with the standard Emacs user interface), just check the help for W3m-Mode via "C-h m".

A Multiple Cursor Trick and an Improvement

I have been baking, in the back of my mind, a way to make multiple cursors more powerful. What I wanted was a way to move through a buffer, and mark certain places where I want to place my multiple cursors, and then make my changes. I think this idea is basically baked enough to reveal that… it is not actually necessary to modify the multiple-cursors library to achieve this goal.

If you want to do this, you can simply insert a sequence of characters that is unique at each point that you wish to place a cursor (something that you would never use, like some crazy Unicode character, say "ಠ_ಠ", you just have to find an easy way to insert it into the buffer). Then, select that tagging sequence and use mc/mark-all-like-this and edit away. This actually works in a pinch. For instance, you can do things like this:

However, I guess that this is slightly more hackish than I like, so I came up with a new, better method. What I decided would be pretty awesome is to have multiple-cursors be able to mark spots off the top of the mark ring (either via popping or just walking the mark ring). I defined a function mc/mark-pop that will just this. I use the suggested multiple cursors and expand region key bindings and bound mc/mark-pop to "C-S-p" (which makes sense on a Dvorak keyboard layout, if you use Qwerty, you might use "C-?"). This means that you can do some pretty awesome stuff like this:

I have submitted a pull request to Magnars which has been accepted. This is still a bit inconvenient to use, but it has promise for being an excellent building block for future multiple cursor editing tricks.

Soft-Semicolons: A little Emacs hack

I have been waging an all out war against the "Shift" key. I find that for programmers these keys, and in particular the left shift key, are used way too much. In my case, this overuse (paired with the common use of control key modifiers and the fact that my keyboard only has a control key on the left side) produced some numbness in my left pinky. This has since been eliminated via removing most of the "shifting" I do on a daily basis by using the Programmers Dvorak keyboard layout. In the process of doing this, however, I realized how annoying and disrupting it is to actually type a shift+key sequence in general. I realized that the annoyance of typing a colon before a symbol is one of the primary reasons that I tend to use the slightly problematic standard symbol notation in Loop or Iterate:

(loop for i below 10 by 2 collect i)

(iter (for i below 10 by 2)
  (collect i))

…rather than the more syntactically and stylistically pure keyword notation:

(loop :for i :below 10 :by 2 :collect i)

(iter (for i :below 10 :by 2)
  (collect i))

So, partly as an exercise in Emacs Lisp and partly just to scratch this personal itch, I decided to modify Emacs behavior in order to make colons very cheap to type. One option is to switch the semicolon and colon on your key map. This makes semicolons more expensive to type, with would be a pretty big loss for C coding where semicolons are much more common than colons. This might lead to the unfortunate situation where your key bindings are not the same between different modes (first layer colons in Lisp, first layer semicolons in C). This is a pretty messy solution.

What I really wanted was to have certain semicolons be converted to colons in certain situations. For instance, if I write a semicolon and then follow it with text (with no whitespace in between), it is very likely that I am trying to write a keyword, so I would like the preceding semicolon to be converted into a colon.

;keyword -> :keyword

But it is certainly the case that if I write a semicolon, then whitespace, then some text, I am trying to write a comment. In this case I want to leave the semicolon alone.

;; Some comment -> ;; Some comment

Naturally, since this is a little hack to save using the shift key, I would like these conversions to be transient, i.e. they only attempt to convert the semicolons if the very next character decides it. For instance, if I move the cursor to the front of some semicolons and start typing, those semicolons should be unaffected (the '_' marks the cursor):

_
;;

;;_

;;some text -> ;;some text

There were a few ways I could think about doing this, but the aim is to be unobtrusive. My solution was to rebind the semicolon key and have it read the characters and commands you give until you give one that isn't "type a semicolon", in which case it decides if it should convert the semicolons it typed on not. This is based very closely on the code in kmacro, namely the function kmacro-end-and-call-macro, which uses the same mechanism to temporarily bind a key (typically "e") to repeat the macro you just performed.

(defcustom *soft-semicolons-also-convert-on* '(9 tab)
  "This variable marks characters that will trigger semicolon
  conversion in addition to the non-whitespace printable
  character requirement.")

(defcustom *soft-semicolons-dont-convert-on* '(?( ?))
  "This variable allows you to exclude certain characters from
  triggering conversion.")

(defun soft-semicolons (arg)
  "Type semicolons like normal expect if they are immediately
followed by a non-whitespace character, in which case convert all
of the consequtive semicolons you were typing into colons."
  (interactive "p")
  (let ((keep-going t)
        (start-point (point)))
    (insert 59)
    (while keep-going
       (let ((event (read-event)))

         (cond ((equal 59 event)
                ;; A Semicolon, insert and keep going
                (clear-this-command-keys t)
                (insert 59)
                (setq last-input-event nil))

               ((member event *soft-semicolons-dont-convert-on*)
                ;; For these special cases, don't do any conversion
                (setq keep-going nil))

               ;; A non-whitespace printable character or something in
               ;; *soft-semiclons-also-convert-on*
               ((or (member event *soft-semicolons-also-convert-on*)
                    (and (integerp event)
                         ;; See if this is a printable character
                         (aref printable-chars event)
                         ;; Rule out whitespace characters (which might also be
                         ;; printable)
                         (not (member event '(9 10 13 32)))))
                (let ((length (- (point) start-point)))
                  (delete-region start-point (point))
                  (insert-char 58 length))
                ;; ...and exit...
                (setq keep-going nil))

               ;; Exit on anything else
               (t (setq keep-going nil)))))

    ;; Push any residual command back onto unread-command-events to be read and
    ;; processed
    (when last-input-event
      (clear-this-command-keys t)
      (setq unread-command-events (list last-input-event)))))

Ironically enough, Common Lisp is one of the only languages I can think of where this soft-semicolon thing interferes with standard syntax. Logical pathname namestrings use semicolons as the delimiter between directories. These directories are necessarily whitespace dependent, and this means that this little hack will make it very annoying to insert logical pathname namestrings. I have never used a logical pathname, I'm pretty sure I never intend to, so I guess this isn't a huge concern for me.

I threw the code up on Github in case you'd like it. This is one of my first forays into Emacs Lisp coding, so I am even more grateful than usual for any comments on how this is implemented or how it could be implemented in a better way.

Adventures in Collaborative Coding With Common Lisp

Update: I've posted a post mortem of the team's attempt this year.

In anticipation of the upcoming ICFP contest (by the way, still looking for team mates, we could use a handful more before I will feel we will saturate our workload), I started looking into collaborative tools for coding. I am aware of a large set of tools that might be useful. This post will describe some these and how we might use them. I am looking at using some subset of Emacs (of course), Slime, Rudel, Mumble, Google+ hangouts, VNC, X11 forwarding (perhaps using XPra), Dropbox, perhaps Git, and naturally ssh to tie it all together.

The basic topology will be like this. I don't know if this helps anybody, but it looks pretty.

Communication between collaborators happens via Mumble and Google+. Google+ has the nice feature that whatever happens in the Hangout will be mirrored to a live Youtube stream and will be saved for future viewing. Files can be exchanged using Dropbox. Rudel allows us to quickly work together and see what others are doing.

The production server is communicated with via Slime/Swank, X forwarding and/or VNC, all via an ssh tunnel. We need X forwarding or VNC in order to make any sort of graphical stuff painless (well, less painful). After experimentation, this is still quite painful. Still looking for a good solution here.

At the end of this post is a pair of videos of a collaborative session I had with one other person on Thursday. The pair of videos are all together quite long and the quality of the video is quite low (much lower than during the actual Hangout), so low that you cannot actually read the text. I'm still trying to figure out how to save the session data well. This was my second attempt to get some kind of example for this post and I felt I couldn't sit on it any longer with the ICFP Contest quickly approaching.

The production server

The first step is to setup a server that can host your Lisp image. This server can really be anything, but you should keep in mind that giving users swank access to a server, is basically giving them shell access at that the Lisp image's privilege level. This means that unless you really trust your collaborators, you should be wary of using a server you care about.

I chose to host off of Amazon EC2 as you only pay per hour of use, so I can start up a fresh system, set it up, and ditch it hours or days later without paying for a month as you might in other places. In a subsequent post (to be posted soon, I hope), I will detail how to set up an EC2 instance for this purpose.

This server will host a Rudel session, a lisp image with a swank connection, optionally a VNC server and Mumble server, and be connected via Dropbox.

Slime/Swank

Most Common Lisp people are probably intimately familiar with Slime and Swank. We are going to be using Slime and Swank to set up a communal Lisp image. Multiple users are going to connect to it and awesomeness will ensue.

We can also have local Lisp images for quicker and/or dirtier work (we don't want to eval broken code on the communal server if we can help it) and anything that needs graphics to run. This is simple and Slime/Swank is ready to go using M-x slime-connect. Use the Slime selector to quickly switch between open connections.

Rudel

Rudel is a collaborative editing library for Emacs. It can use many backends, but we will be using the Obby backend as that was the only one that was easy to set up. Once Rudel is loaded, one person can host a session, which is then joined by any number of participants. We will be hosting the rudel session from the production server. Buffers within Emacs can then be published by one party and subscribed to by any number of other people. After this, that buffer on each computer will hold the same contents, updated in real time as the people code. Text edited by a particular user will be marked in his/her specific background color. There are some packages you will need:

apt-get install emacs23 gnubin-tls avahi-daemon avahi-utils

Setting up Rudel is easy so long as you get the correct version (the one from SourceForge). Once you download and extract it, just add this single line to your .emacs file.

(load-file "~/.emacs.d/rudel-0.2-4/rudel-loaddefs.el")

Note that Rudel uses "C-c c" as a prefix command, which is weird to me, so if you use "C-c c" for anything, either remove that binding, or bind that after you load Rudel, so you can effectively clobber their bindings.

With a couple exceptions (see below) Rudel is pretty painless to use, just join the session, subscribe to some buffers or publish your own, and start editing. It is a good idea to have your Rudel session hosted by an Emacs instance on the production server (so you don't have to kill your Emacs to reset any problems). This is also a good idea just so your computer isn't the single point of failure for the team. You can go to sleep and shut down your computer without effecting others.

One issue for lisp programming is that you can't share the REPL buffer (slime-repl-mode can't be simply turned on and off, nor can you insert text into it all willy-nilly like Rudel assumes it can). However, you can share a Slime scratch buffer, or any buffer that is in slime-mode, which is basically just as good. Google+ allows you to make more involved presentations at the REPL between collaborators if that is needed.

One annoying thing about Rudel is that it seems to be impossible to actually leave a session. When you attempt to leave the session via rudel-end-session, you are disconnected and unsubscribed to all of the buffers, but your login remains and the server keeps the connection open (I believe). This doesn't seem too bad until you try to join again and realize that you can't because your username (and possibly color) are currently in use. To get around this, I just append a number on the end of my username in order to make it fresh every time. Regarding colors, I just pick a garish one when logging in and then change it to something better once I have joined (once you have joined, you can have the same color as someone else). Most likely, most people will work with colors turned off anyway.

Another annoying but (logically consistent) feature of Rudel is that M-x undo will undo other peoples edits as well as you own. This is something which is sometimes desired, but often times not if there are two writing code concurrently. If kill-undo is burned into your muscle memory instead of kill-yank, then you might have some problems. I am trying to come up with a work around for that particular case. Other times can be handled by simply specifying the region and using undo within that region (see the undo help page).

Git, Rudel, and Dropbox

The summary of this section is that these tools don't work together, at all, at a fundamental level. Use Rudel. People can use Git on a person by person basis. Just give up on the idea of sharing a source directory via dropbox. It is a lost cause to try combining Dropbox, Rudel, or Git simultaneously. Be warned that Git will be crippled when using it this way, you can't do any of the good git stuff like branches, reverts, and merges, as it will mess up everybody else's Rudel buffers.. Always unsubscribe, do any fancy Git commands you like, then republish (possibly under a different name) or subscribe and replace the entire file (presumably with the approval of the people sharing the file).

While I have never participated in a short dead line contest and actually used a version control system, I am sufficiently sold on the idea of distributed version control that I would like the option of using it here. Using a version control system has kind-of been integrated into how I think development should be done in general. Development should be broken into smaller, separable tasks which should be made as commits with commit logs telling the future developers (a.k.a. you) why this was done. Developing without it would make me feel naked, or at least haphazard.

That said, there is a problem with using git and Dropbox at the same time. In fact, it is a very fundamental problem. Dropbox is attempting to make two or more directories seem to be the same, no matter what computer you are on. Git, on the other hand, explicitly works under the assumption that the directories are on two different computers and are absolutely independent.

For an example of this conflict, consider a group of people collaborating on a project using Dropbox. As one person edits files on his computer, these edits are quietly sent to the other computers (which causes you to have to constantly revert buffers in Emacs and leads to conflicts, but let's say we are okay with that). If you are using Git, any change to the repo will also be synced. This seems good at first, until you realize that the index is in the repo. This means that you can't develop like Git wants you to develop, incrementally building up the index, crafting your commit, then committing. Each developer would step on the others' toes as they add to the index. Instead you need a process where you build your index in your head, then put a freeze on development to commit, e.g. "hey, nobody do anything, I am going to commit something." This basically eliminates most of the positives that Git brings to the table.

I tried several schemes of moving the .git directory out of the Dropbox folder which fixes this whole index problem. When you do this, you get back all of that Git goodness, but you lose the idea of a synced repo, so why use Dropbox at all? In fact it is worse that that. You now have two conflicting ideas of what the merged repo will be. You cannot combine the two, only discard one and accept the other. So, I submit, that Dropbox and Git just do not mix for this purpose. It can't be done in a sane way.

Everything I just described regarding Git and Dropbox is also true of Git and Rudel. Rudel, however, comes with the extra limitation that you can't just change files on disk anymore. The buffers might be saved to your disk, but the real buffer is in the "cloud". So, changing the file on reverting to a file on disk will break Rudel. From what I have seen, your buffer will no longer be in sync with others. It is important to note that you could actually reconcile this limitation by replacing revert-buffer with something that edits the changed lines in a Rudel approved fashion. But right now, this is not supported.

I lean towards using Rudel as I feel it is more important. We will still use Dropbox for easy file transfer, and people can still try to use git so long as they don't attempt to use any commands that will change the buffers on disk. No one should try to concurrently develop in the same synced Dropbox folder, though. I might setup a backup script that will run every 5 minutes or something and take a snapshot (using rdiff-backup, for instance) of my files so that we can roll back to previous versions if everything hits the fan.

Another thing that can be done with Git is to have a single person in charge of version control of a given file. That person will watch what other people are doing and make commits as needed. That person also institute reverts, branches, and merges, but such actions really need to be done via a safer mechanism than a simple git-checkout or git-merge. The person in charge of version control should really unpublish the file and republish it once the change is made.

Google+ and Mumble

I really like Google+ hangouts, they are about as close sitting at the same desk as someone else as video chat has ever gotten. As nice as Google+ is, it is a bit annoying to have to leave that CPU/network hog running non-stop in order to communicate. This can be partially handled by Mumble, a VoIP push to talk program. It is light weight and can be left running non-stop without many issues. I'm not sure what will be better in practice.

VNC and X Forwarding

VNC has a lot of issues when it comes to connecting two peers, particularly two peers that might be using an ISP that won't allow incoming connections from the Internet. It will work well with a one computer acting as a server that is accessible from the Internet. However, ever when you have VNC working, you can usually count any OpenGL out of the equation. Attempting to use OpenGL on EC2 resulted in a crash of the Lisp system, if I remember correctly. Replacing the OpenGL drivers with a software renderer might help (in fact it seems necessary as the EC2 server has no video card for a hardware driver to make sense). But the main issue is the lag, which is pretty bad, and the general frame rate. However, you can certainly setup a GUI with buttons, combo boxes, static images, etc, and it will work fine. The only real issue is real-time graphics.

X forwarding is another option, it can be made pretty efficient with the help of XPra or NX, and with XPra, at least, the window can be detached and re-attached by someone else. But this is not really collaborative, though. To my knowledge, there is no way to use this technology (or technology like it, e.g. XPra or NX) in a collaborative way. If you do choose to use it and you are using EC2, I could only get it to work if I installed the software rendering drivers (otherwise the Lisp system crashed, this is actually not that uncommon if you are using CL-GLUT).

The Experience

This is a really neat experience. Rudel and sharing a Lisp image was a very new experience to me, and it felt like there was a lot of potential there. It will take me, and probably others, some time to actually wrap my head around all of the implications here. I often times found myself forgetting that I can edit the buffer while someone else is editing something else, or that I can evaluate that code and it will instantly become available to the other users. I can only imagine that this parallel development could scale nicely with more people. You do need to coordinate the development, but this is always true. Problems need to be broken into distinct, separable subtasks and the solutions to those subtasks need to have well defined interfaces in order to prevent breaking other peoples code, but this is, again, always part of any development with more than one person. There are also times where it is very clearly a win, for instance, when writing unit tests or debugging at the REPL.

Of course, if you are really interested in playing with this, hopefully this post and optionally my subsequent post on setting up EC2 will allow you and your friends to try this out yourself. Also, I'd once again like to put in a plug for my ICFP team, join us, it will be fun. Beyond that, at least for the time being, I will put out a standing offer that if you want to have a collaborative coding session with me, in Common Lisp, let me know and I will probably be happy to participate.

Here are the videos of the coding session. Again, I apologize for the quality and the slowness of the development (Oleg and I are still learning each other's style). The task we were setting out to accomplish was to design a program that could solve a maze.

Other things we could have used

There are tons of tools out there. I am aware that you can do a lot with communal Screen sessions if you are willing to limit yourself to the terminal. You can also just run Emacs over X11 forwarding (Emacs has the capability to spawn frames on different displays). This might work, but some have said that Emacs can freeze if one of the users drop their connection (I suppose without closing the window).

If anybody knows of any other awesome tools, or a better set up like this, please comment below, I'd love to hear about it.

Ubuntu 12.04 vs. Emacs Key Bindings

As I am becoming more at home in the Unity interface I came across the annoying problem that Unity binds the <Control><Alt>t chord for starting a terminal window. While that is a fine chord for that, and I believe that starting a new terminal should be as easy as possible, that key-binding is already used for something I use even more frequently, transpose Sex-Ps in Emacs.

In the past this could easily be rebound by using ccsm, the CompizConfig Settings Manager, and entering the Gnome Compatibility plug-in and setting it to something else. Unity (at least what comes with 12.04) ignores this binding, it seems. In fact it seems that there are several places where this binding might be set. I remembered stumbling upon a list of bindings in MyUnity, or UbuntuTweak, or some other third party tweaking app, but I have long since forgotten where that was. But using every place I found, disabling that key-binding never had any effect.

I finally took the time to work out a solution today. The solution is to get my hands dirty and use gconf-editor directly. I don't think gconf-editor is included in the default install of Ubuntu, so you need to:

sudo aptitude install gconf-editor

Start the program and search for keys that have the word "terminal" in them. I found three places where that key-binding was specified and I wiped out the value in each, though it was the first key value that seemed to matter. Then if you wish to set a binding, run CompizConfig Settings Manager and edit the key binding to start a terminal under the Gnome Compatibility plug-in. Once again, CompizConfig Settings Manager doesn't come installed by default, so:

sudo aptitude install ccsm

I don't think that Canonical is really interested in promoting deep customization of their OS or window manager, something that is not very GNU/Linux or Libre Software like. This is a very different zeitgeist from the GNU/Linux of a decade ago, or even five years ago. Maybe that is why I had such trouble changing this binding. How is this not a bug that would have been fixed in 11.04? It's all fine, though, so long as they don't take that extra step of actually obstructing people from customizing things.

Update: I recently reinstalled Ubuntu 12.04 and used none of the old configuration files in the new install. After this, setting the shortcut under Settings -> Keyboard -> Shortcuts (tab) -> Launchers does the correct thing. No need for gconf-editor or ccsm or anything else.

Cedet Interferes with Slime

Just in case this has bitten others: Cedet (at least the current 1.1 version) does not play well with Slime. It clobbers some Slime facilities, like arg-list documentation in the mode line, and it seems to map capital letter key bindings to lower case key bindings (like C-c C-d A seems to run C-c C-d a which is slime-apropos). There are probably other annoying "step on other libraries' toes" sorts of bugs as well. My advice is that if you are seeing odd Slime behavior, or odd anything behavior, pull the Cedet stuff out of your .emacs file and try without it.

This almost certainly has to do with the way many Cedet setup guides direct you to enable a bunch of global-xxx-mode settings. However, I don't have any lines like this in my .emacs file and still see odd behavior, so I wouldn't be surprised if this is the default.

I found this that briefly sketches how you might only locally enable Cedet. This seems a bit involved and there is no guarantee of it resolving the problem. So, I guess I am back to using by memory to figure out structure members like some kind of animal.

On LaTeX and Microsoft Word

Sorry, this is a ranty post. Adjust your desire to read appropriately.

I was busy writing my APS abstract this last week. When we got near the end I was having a heck of a time getting Bibtex to work with Latex and insert my references properly. Later on I realized that APS doesn't really allow for citation ala Latex and Bibtex, but whatever. I was struggling with this, and mentioned to my advisor, jokingly, "Man, I hate Latex." He says, also jokingly, "you could use Microsoft Word." I say sure, despite the fact that the APS website says they prefer Latex and that Latex is actually easier for the submitter, MS Word is no problem. In fact, that is kind of the point. I wrote my abstract using Org-Mode precisely because it made little to no assumptions on the end result. I can export to Latex or one of the many other formats or just grab my text and paste it into MS Word with no problems. I can also send it to any human being in the world with a computer and that person can open it and understand what they are seeing. Hell, in a handful of keystrokes I could post it to this blog.

Later, when printing, my advisor changes from his joking suggestion to insist that I use MS Word (presumably so that I can export to a Rich Text Format document that the APS accepts). This will make it easier, right… So, I copy/paste it over to a document and save it as an RTF file. Low and behold, of course, the equations are not interpreted by word. It would take at least 15 minutes and perhaps up to an hour to figure out how to get the proper symbols, fonts, and kerning into the document for submission. The last kick to the nuts is that the APS provides an RTF template, but in order to use it, you must actually type the document into it yourself, by hand. If you copy and paste it into the template something, apparently, will get screwed up. I will repeat that, the American Physical Society requires you to actually, physically, press the keys on your keyboard while their template is open in order to submit an abstract (though I imagine something like xdotool might serve me well here). This is what happens with you use things you don't understand people. You get black magic crap like this.

Point is, I miss my previous colleagues and advisor, nay entire department, nay entire institution where they held the opinion that any individual in the sciences should be using Latex as their format for correspondence. Also, bravo to the APS for encouraging people to use Latex for submitting abstracts.

Fixed Point Emacs Completion

One of those annoying things about GNU Emacs is that when a window is popped up containing completions, it derails what you were doing.

First, completion windows take up half the frame. I can't actually figure out how to change this easily. Think of how you use the completions window. One primary way I use it is: I try to complete a symbol, realize that it doesn't have enough information, then type more and try again. In this case, having so many options doesn't help at all. I only need to know if there the symbol can be completed or if there are multiple possibilities.

Second, the position of the cursor on the screen is almost always changed when a window is popped up, and interestingly enough, not even in a predictable way. This means that if I try to perform a completion which I expect to have only one completion, thus not popping up a window, but it actually has several possibilities, I have an annoying situation. This could happen as a result of a symbol I didn't know about making the completion ambiguous resulting in a pop-up and a partial completion or it could be due to a typo. In either case, I need to examine what is at the command prompt in order to find out the proper course of action. This is unavoidable. What is avoidable, however, is the need to move the command prompt around the screen forcing the user to go searching for the new location.

In addition to these pretty legitimate claims, I also just find the entire process to be jarring. I often find myself kicked out of my current train of thought as I search for where my cursor has gone. Here is an example of how completion tends to work in Emacs.

When these two effects combine, I got so frustrated I started leaving my frames split in half so that completions will show up in the other window without effecting the window I was editing in at all. In a recent Reddit conversation with Stassats, he states that he also always maintains a completion window. To me, this is a big waste of screen space, but at least the point doesn't jump.

There are other solutions to this problem. My guess is that many people struggle with this and each produce their own solution. For instance, here is a post of how to designate a particular window for completions in someones (rather complicated for my taste) completion window setup.

My solution

The code is here, though I can't vouch for it being bug free.

My solution is to have windows pop-up with completions but have those windows be smaller than half the screen. An important distinction is that when a completion window pops up, it will not alter the screen location of the point. In order for this to be the case, we need to pop-up a window in an area of the frame that the cursor isn't utilizing. So, we handle this by identifying where in the frame the point is, then popping up a smaller completion window in either the top or bottom of the frame depending on its location.

To be precise:

1. See if there already is a visible completions window. If one exists, use it.
2. See if the current frame has only one window (when not counting the minibuffer). If it does, we know that a popup is desired.
   1. If the current window is actually the minibuffer, create the popup at the bottom of the frame.
   2. If the point is above the halfway point in the frame, create the popup window at the bottom of the frame.
   3. If below the halfway point, create it at the top of the frame.
3. If it matches none of this, let Emacs do as it would have.

Here is the result:

This works pretty well for me, but I imagine that is just because I might use Emacs in a quirky way. For instance, I usually have one desktop where all of my Emacs windows are. If they are on different desktops, perhaps the "use completions window if 'visible'" won't work very well. I don't know how Emacs handles this. What about terminal emacsclient instances and how does completion there interact with open X windows. All of these things should be configurable.

Anyhow, if anybody else finds this useful and wants to push patches back to me, please do. If anybody wants to grab this code and take over the project, or include it in your project, again, please do.

Other completion extension libraries

While I haven't found anything out there that does exactly what Fixed-Point-Completions does, there are a few libraries that also tweak completion. Most notably, popwin and icicles extend the completions functionality. Icicles is a bit heavy handed for my taste, changing many things in how Emacs functions. Popwin does a lot towards fixing my completion issues, but not enough. The thing which is really missing from both of these is that they still shift the visual location of the point at times.

Emacs Pinky

Whelp, it has happened to me.  After switching to Emacs 2-3 years ago, I have finally developed a repetitive stress injury due to my programming.  I don't think this would have happened with Vim (my old editor), but I have been coding much more these days.  Whatever.

I am resistant to using Viper mode as I think it will mess with the key bindings in the 20 some odd major modes I use every day.  So I am going to try something else.  I am deleting the Ctrl and Shift modifier keys from the X key bindings on the left side of my keyboard and will force myself to use Shift and Ctrl on the right hand side for a while.  Spreading the wear out over both hands should help things a bit.  Of course things are complicated a bit by the fact that this Mac keyboard doesn't have a left control at all.  No biggie with xmodmap.  Just use xev to figure out what keys send what code and make a xmodmap input file like this.

clear control
clear mod1
clear mod4
clear shift

keycode 50 = Shift_L
keycode 62 = Shift_R
keycode 37 = Control_L
keycode 108 = Control_R
keycode 64 = Super_L
keycode 134 = Alt_R
keycode 133 = Alt_L

add shift = Shift_R
add control = Alt_R
add mod1 = Alt_L
add mod4 = Super_L

Oh yeah, if you are using Ubuntu or probably any mainstream distribution, changes to xmodmap are often times intercepted on startup and different key mapping utility is used. in short, make this file as xmodmap in your home space, apply it, then restart the X server. You will probably get a dialog asking you if you want to include these key mappings or not.

Just run xmodmap <input-file> to effect the changes.  Also, I am going to try and mimic the key placement of the older space cadet keyboards, which were made for Emacs.  The Wikipedia page states that a big difference was that the space cadet keyboards had modifier keys in a "thumbable" position.  So I am leaving my left Alt as Alt and setting my right Alt as a Ctrl.  This means that in a few weeks time I should be using thumbs for most Emacs work (and everything work, as xmodmap changes are X global).

Now, the only thing I use my left pinky for is typing and Tab, which seems fair.  I'll see if things get better.  Maybe I will end up in Viper mode.

Update: Things did get better, but just barely. I do believe that the numbness is caused primarily by just a few common Emacs commands, like C-c C-c, C-x C-s, or any command that uses the C-c or C-x prefixes. In fact, my opinion is that the left hand is way over used in Emacs. I had to re-enable the left shift key for the sake of my typing speed. This has also revealed that Apple cut a few corners when it came to their keyboard hardware (everybody does). The left and right alt keys, when pressed at the same time, will stop any keys on the "zxcv" row from even producing key codes, so that's a pain right now. As of testing just a few minutes ago, the numbness comes back with just a few minutes of typing on a normal keyboard layout.

Update (after a few months): This has basically been resolved.  A few thoughts: mapping to thumbs seemed like a good idea, but thumbs get worn out too.  After a few months, my thumbs were pretty ache-y.  In the end, the two things I found that really did make a difference were: 1. binding caps-lock as a control and 2. switching to a Dvorak layout, more specifically the Programmers Dvorak layout, which I think is pretty smart.  I can't say how much each of these contributed individually, as I changed them simultaneously.  I plan to write a post on Programmers Dvorak soon but suffice it to say, switching layouts isn't a decision to take lightly.  The caps-lock thing, on the other hand, is dead simple and helps tremendously; I very much suggest it.  I only wish I had another key right beside the caps-lock (or in the pinky position on the right hand) that I could bind to Meta/Alt.

Syncing Emacs with Google Documents

Before anything, sorry for the long post. If I had more time, I would have made it shorter. I made a promise to myself that I would spend the time to write up things I do, mainly so I get practice doing so, and to have a record of things I've done.

Recently I had an interview with Google. At Google they try to weed out applicants with a quick technical phone interview before they fly you out for an actual interview. As part of this interview, a shared Google Document is set up as a kind of virtual white-board so you can show your programming ability. Never mind that I would rather be sharing a Screen session, VNC connection, or even a live screen cast of my desktop, this is the way they decided to do it. The problem is, Google Documents are not meant to be code editor, they are meant to be a tool for writing documents. As such, I would be left without the aid of auto-completion, argument lists, and (most importantly) the Emacs key bindings, that, by now, are etched into my spinal column. So, a few minutes after receiving an email stating a Google Document would be used, I started coding up a program that would allow me to program in Emacs, but have my work displayed in Google Documents. See the video for an idea of what I'm talking about.

I quickly found the hooks: before-change-functions and after-change-functions. Actually, they are not hooks, I believe, as Emacs has a strictly specified meaning and argument passing protocol for hooks, they are just a list of functions that are called before and after each change, respectively. These would allow me to find what changes are being made locally so I can send them to Google Docs.

Now came one hurdle: how do I interface with Google Docs? There is a Greasemonkey and Python script that allows you to download Google Documents from the command line. Unfortunately, I could not find a script made for uploading documents, which is really the main part of this. Further, while the script worked for downloading, it was far too slow for interactive editing. The time from hitting return to retrieved document was on order 5-10 seconds. I needed something that could be run extremely fast, like on the order of the time it takes to type a key.

I eventually decided that I should try to mirror the edits between the two document by sending keystrokes to the browser window. Next hurdle: in X, as far as I know, keystrokes are only sent to the focused window. And since I would be using Emacs, I couldn't rightly send keystrokes to Firefox at the same time. Or can I? I quickly decided to run a separate X server which would run an instance of Firefox which could be focused at the same time as Emacs on my normal X server. Now there are several ways to do this (e.g. actually run a new server on a different virtual terminal, or playing around with Xnest which runs an X server inside a window in your host X server) but in the end I decided to use the awesome programs xpra and Xvfb. Xpra is a program designed to be an alternative to X forwarding.

Xpra and Xvfb

As an aside, X forwarding allows you to run graphical applications on remote machines while having the graphics display locally. This is different from VNC and RDP in that you don't have access to the entire X session, just windows associated with applications you are running. X forwarding has become less used in the Unix world, mostly due to a lack of knowledge of its existence, I believe. This is not to say that there aren't good reasons for it not be used. It actually has a pretty severe problem on high latency network (a.k.a high ping time) connections.

Xpra was also designed to get around X forwarding poor performance as well as to add abilities like detaching GUI programs and re-attaching from a different point on the network, just like with screen. It truly is very neat. All of it's neatness aside, I just wanted to use it to handle Xvfb, which is a virtual X server, like Xnest, but one that doesn't have to be displayed at all. You can start programs under an Xvfb server and they will happily do their business thinking they are displaying windows, but no window is drawn to any screen. Xvfb sounds just like what I needed. First I started the Xvfb server on display ":1" by using the Xpra:

xpra :1

I started Firefox under the Xvfb server via:

DISPLAY=:1 firefox -no-remote -P some-other-profile

This is more complicated than normal because I leave Firefox running. The options -no-remote tells Firefox to not check for other, currently running instances of Firefox before starting it. If Firefox sees a running instance of itself, it will choose to just open a new window rather than start a new instance. The option -P tells Firefox to use a different profile (which you must create prior to this) for the session. This is necessary as a different running instance of Firefox will have a lock held on your normal profile. If you are willing to just close Firefox, you can omit both of those options.

At this point, nothing should be displayed. If you want to attach the Firefox window to the current X server via Xpra, just use xpra attach and it should appear. You can navigate this to your Google Document.

Sending input to the window

Up until now, we have not even discussed how you are going to send input to the window. Luckily, this can be accomplished by many tools. In fact, I ran into three without even looking really (Xnee which includes gnee and cnee, xte, and xdotool). I have even heard people describing directly inserting characters into the input devices in /dev. I ended up using xdotool because (1) it seemed more powerful than xte and (2) cnee crashed my (main ":0") X server. With xdotool you can send keys (with modifiers), strings, mouse motion and clicks, and manipulate windows, like bringing them into focus.

# Send an 'a', a 'B', and a 'C'
DISPLAY=:1 xdotool key a B shift+c

# This types "hello how are you"
DISPLAY=:1 xdotool type "hello how are you"

# This moves the mouse to 100,100 relative to the top-left corner of the window,
# then right click (presses and releases the 3rd button)
DISPLAY=:1 xdotool mousemove --window 41466948 100 100 click 3

# This brings window with id 41466948 into focus
DISPLAY=:1 xdotool focusWindow 41466948

This is fine and dandy, but what if we want to input large sections of text. Even if we use the type command, xdotool is still just typing every letter, albeit ~100 characters/second. Further, due to a bug/shortcoming/whatever in xdotool, I couldn't figure out how to make it send text that includes a newline character (or an ampersand or question mark for that matter). It seems worth our while to come up with other ways of interfacing with the Google Documents window. The other way I came up with is the clipboard.

X has a clipboard (actually it has three), and that clipboard can be accessed via the command line. Again, there are multiple tools that can do this including xclip and xsel. While the functionality of the two programs are near identical, I chose to use xsel as it seemed a little more robust. Using xsel, we can send large sections of text to the clipboard on display ":1" (remember, each X server has it's own clipboard), then send a "ctrl+v" using xdotool. More interestingly, however, is that copying in the ":1" server allows us to gain some information on what is in the other file and more importantly, as it turns out, the location of the point in the Google Document.

Putting it together

Most of this involves coding in Emacs Lisp. The point of everything here is to keep track of where the point in the Google Document while making alterations. If we ever lose track of where the point is in the document, we will corrupt the document with every edit. In the following, I use the word "document" to refer to the Google document and "document point" to refer to the cursor location in that document. The word "buffer" refers to things in Emacs, and "buffer point" refers to the point in the Emacs buffer, i.e. the cursor position. I use the word function here because that's what Emacs calls them. In reality these are all procedures called for their side effects.

We will need some basics:

1. A function that will send a key to the window
   
2. A function that will send a string to the window
   
3. A function to properly escape strings sent through the shell
   
4. Functions that move left and right
   
5. A function that moves the point by some relative amount
   

   (defvar gd-point 0 "A variable holding the document point")
   
   (defun gdmir-gdocify-string (str)
     "Escape every character.  This means that the shell shouldn't
   mess with any of this."
     (replace-regexp-in-string "\\(.\\)" "\\\\\\1" str))
   
   ;; The parameter INSTANT is used in the buffered input optimiztion, but not
   ;; here.
   (defun gdmir-send-key (keys &optional instant)
     (shell-command (concat "DISPLAY=:1 xdotool windowFocus 4194333 key "
                            keys )))
   
   (defun gdmir-send-string (string)
     (shell-command (concat "DISPLAY=:1 xdotool windowFocus 4194333 type "
                            string )))
   
   (defun gdmir-move-left (n)
     "Move the buffer N characters left, reducing GD-POINT by N.
   The buffer point doesn't change."
     (loop repeat n
           do (gdmir-send-key "Left")
              (decf gd-point)) )
   
   (defun gdmir-move-right (n)
     "Move the buffer N characters right, increasing GD-POINT by N.
   The buffer point doesn't change."
     (loop repeat n
           do (gdmir-send-key "Right")
              (incf gd-point) ))
   
   (defun gdmir-move-to-relative-point (point-difference)
     "Move the buffer point to \(+ GD-POINT POINT-DIFFERENCE).  The
   buffer point doesn't change."
     (if (< point-difference 0)
         (gdmir-move-left (abs point-difference))
         (gdmir-move-right (abs point-difference)) ))
   

Now for the meat and potatoes: the hook functions. We will define a function that sets the after and before change functions. The before function, gdmir-before-edit, which receives the start and end point of the forthcoming edit, moves the document point to the end of the edit range, then saves the original string, and finally deletes the original text¹. The after function, gdmir-after-edit, sends commands required to type the new text.

(defun split-for-xdotool (string)
  "This function runs through the input and replaces instances of
`?', `&', and newlines with symbols corresponding to these
characters.  We return a list containing strings and the special
symbols in the order they should be output.  We have to use the
special symbols due to the fact that xdotool, or bash, or
something cannot handle these symbols, even if they are quoted."
  (rest
   (let ((count 0))
     (loop for segment in (split-string string "[\n?&]")
           appending
        (list (when (> count 0)
                (cond ((eql (aref string (- count 1)) (aref "?" 0))
                       'question )
                      ((eql (aref string (- count 1)) (aref "&" 0)) 'amp)
                      ((eql (aref string (- count 1)) (aref "\n" 0))
                       'ret )))
              segment )
           do (incf count (1+ (length segment))) ))))

(defvar *prechange-text* nil "This holds the contents of the
edited region prior to the edit.  We don't use this, but I can
think of a few reasons we might want to save it." )

(defun gdmir-before-edit (start end)
  (gdmir-move-to-relative-point (- end gd-point))
  (setf *prechange-text* (list start end (buffer-substring start end)))
  ;; Delete the string in GD before in Emacs
  (loop for i below (- end start)
        do (decf gd-point)
           (gdmir-send-key "BackSpace") ))

(defun gdmir-after-edit (start end old-length)
  (when (< 0 (- end start))
    (loop for line in (split-for-xdotool (buffer-substring start end))
          do (cond ((eql line 'question)
                    (gdmir-send-key "shift+slash") )
                   ((eql line 'amp)
                    (gdmir-send-key "shift+7") )
                   ((eql line 'ret)
                    (gdmir-send-key "Return")
                    (gdmir-send-string "\\|") )
                   ((< 0 (length line))
                    (gdmir-send-string (gdmir-gdocify-string line)) )))
    (incf gd-point (length (buffer-substring start end))) ))

(defun insert-change-hook ()
  (setf *change-hook-in-effect* t)
  (push
   'gdmir-before-edit
   before-change-functions )
  (push
   'gdmir-after-edit
   after-change-functions ))

We also can use a few helper functions which we will implement as key-bindings.

1. A function that syncs the points in the buffers (C-x SPC when mirroring is on)
   
2. A function that turns the mirroring on and off (C-x SPC when off will turn
   it on, if it's on and the point is already synced, it will be turned off)
   
3. A set of functions that moves the point in the browser window (M-<direction
   keys>). These are really handy for a quick reposition of the document
   point.
   

   (defun sync-points ()
     "Set GD-POINT to the buffer point."
     (setf gd-point (point)) )
   
   (defvar *change-hook-in-effect* nil "Used to determine if our
   change hooks are already doing their thing.")
   
   (defun setup-mirror ()
     "Set up the mirroring environment.  This should really be a
   minor mode, but since this is basically a throw away hack, I'm
   not going to bother."
     (setf old-after-change-functions after-change-functions
           old-before-change-functions before-change-functions )
     (local-set-key (kbd "M-<left>")
                    (lambda () (interactive) (gdmir-send-key "Left" t)) )
     (local-set-key (kbd "M-<right>")
                    (lambda () (interactive) (gdmir-send-key "Right" t)) )
     (local-set-key (kbd "M-<up>")
                    (lambda () (interactive) (gdmir-send-key "Up" t)) )
     (local-set-key (kbd "M-<down>")
                    (lambda () (interactive) (gdmir-send-key "Down" t)) )
     (local-set-key (kbd "C-x SPC")
                    (lambda ()
                      (interactive)
                      (cond ((and after-change-functions
                                  *change-hook-in-effect*
                                  (= (point) gd-point) )
                             (message "Mirroring disabled")
                             (setf *change-hook-in-effect* nil
                                   after-change-functions old-after-change-functions
                                   before-change-functions old-before-change-functions ))
                            ((and after-change-functions
                                  *change-hook-in-effect* )
                             (message "Syncing points")
                             (sync-points) )
                            (t
                             (message "Mirroring enabled")
                             (setf *change-hook-in-effect* t)
                             (insert-change-hook)
                             (sync-points) )))))
   

Optimizations

While all of this works, it works quite slowly. Just moving from one point to another in the document can take a considerable time as it involves moving side to side perhaps thousands of times. Here we consider two areas where there can be considerable improvements.

Incorporating other moves than side to side

The first optimization I made was to allow for motion using the up and down keys. This is difficult as we don't know how the lines are wrapped on the Google Doc, so we don't know how far that moves in the file. This can be attacked by noting that if we hold down shift as we move, the region will be selected. We can copy that to the clipboard² and read it in Emacs, allowing us to find length of the string selected, and therefore the motion in the document. This almost works. However, it turns out that whitespace at the beginning and end of selections are often times elided when copied to the clipboard (not sure why). Another issue, if you make a selection of only whitespace, it may not be copied to the clipboard at all, leaving what was on there before.

This means that we may lose track of our document point if we start or end on white space in a vertical move. The way I chose to deal with this is to place a vertical bar, "|", at the beginning of each line. To move vertically, I move the point to the beginning of the line (before the vertical bar) and then move up or down selecting the difference. This method ensures that the selection includes no end whitespace (other than the newline at the end. The newline at the end of the selection is deleted but it is compensated for by the vertical bar. As long as we only move one line at a time, we can tell how far we've gone in the vertical direction.

To add this change, we must modify the gdmir-move-left and gdmir-move-right functions to take two steps when we pass a newline in order to skip over the vertical bar in the left hand column. We must also have our change hooks check to see if we are deleting a newline (we must delete an extra character) and if we are inserting a newline (we must add a new vertical bar).

(defun gdmir-move-left (n)
  (save-excursion
    (goto-char gd-point)
    (loop repeat n
          do (when (= 0 (current-column))
               (gdmir-send-key "Left") )
             (gdmir-send-key "Left")
             (decf gd-point)
             (backward-char) )))

(defun gdmir-move-right (n)
  (save-excursion
    (goto-char gd-point)
    (loop repeat n
          do (gdmir-send-key "Right")
             (incf gd-point)
             (forward-char)
             (when (= 0 (current-column))
               (gdmir-send-key "Right") ))))

(defun gdmir-before-edit (start end)
  (gdmir-move-to-relative-point (- end gd-point))
  (setf *prechange-text* (list start end (buffer-substring start end)))
  ;; Delete the string in GD before in Emacs
  (save-excursion
    (goto-char gd-point)
    (loop for i below (- end start)
          do (progn
               (when (= 0 (current-column))
                 ;; Clear out code marker
                 (gdmir-send-key "BackSpace") )
               (decf gd-point)
               (gdmir-send-key "BackSpace")
               (backward-char) ))))

(defun gdmir-after-edit (start end old-length)
  (when (< 0 (- end start))
    (loop for line in (split-for-xdotool (buffer-substring start end))
          do (cond ((eql line 'question)
                    (gdmir-send-key "shift+slash") )
                   ((eql line 'amp)
                    (gdmir-send-key "shift+7") )
                   ((eql line 'ret)
                    (gdmir-send-key "Return")
                    (gdmir-send-string "\\|") )
                   ((< 0 (length line))
                    (gdmir-send-string (gdmir-gdocify-string line)) )))
    (incf gd-point (length (buffer-substring start end))) ))

(defun insert-change-hook ()
  (setf *change-hook-in-effect* t)
  (push
   'gdmir-before-edit
   before-change-functions )
  (push
   'gdmir-after-edit
   after-change-functions ))

Then we can introduce gdmir-move-up, gdmir-move-down, and extend gdmir-move-to-relative-point so it makes use of this new ability.

;; *EMS-PER-LINE* will be used to guess at when a line might be wrapped.  if we
;; *over-estimate, vertical moves will be broken
(defvar *ems-per-line* 50
  "A conservative estimate for how many `m's are in a line in the google document." )

(defun gdmir-move-to-zero-column ()
  (flush-commands)
  (save-excursion
   (goto-char gd-point)
   (gdmir-move-left (max (- (current-column) (- *ems-per-line* 1)) 0))
   (goto-char (- gd-point (max (- (current-column) (- *ems-per-line* 1)) 0)))
   (cond ((< (current-column) *ems-per-line*)
          (gdmir-send-key "Home Right" t)
          (setf gd-point (line-beginning-position)) )
         (t (gdmir-move-left
             (current-column) )))))

(defun gdmir-grab-selection ()
  "To read from the clipboard"
  (gdmir-send-key "ctrl+c" t)
  (shell-command-to-string "xsel --display :1 -o -b") )

(defun gdmir-move-up ()
  (flush-commands)
  (save-excursion
    (goto-char gd-point)
    ;; More to the left most position on the screen
    (gdmir-move-to-zero-column)
    (goto-char gd-point)
    ;; Move to the left of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left")
    ;; move up selecting the differnce
    (let ((orig-line (line-number-at-pos)))
      (loop until (/= (line-number-at-pos) orig-line)
            do (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key shift+Up")
               (let ((selection (gdmir-grab-selection)))
                 (backward-char (length selection)) )
               ;; This doesn't move the point, it just moves to the left of the
               ;; selection and unselects the text.
            (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left") ))
    (setf gd-point (point))
    ;; Move to the right of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))

(defun gdmir-move-down ()
  (flush-commands)
  (save-excursion
    (goto-char gd-point)
    ;; More to the left most position on the screen
    (gdmir-move-to-zero-column)
    (goto-char gd-point)
    ;; Move to the left of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left")
    ;; move down selecting the differnce
    (let ((orig-line (line-number-at-pos)))
      (loop until (/= (line-number-at-pos) orig-line)
            do (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key shift+Down")
               (let ((selection (gdmir-grab-selection)))
                 (forward-char (length selection)) )
               ;; This doesn't move the point, it just moves to the left of the
               ;; selection and unselects the text.
            (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))
    (setf gd-point (point))
    ;; Move to the right of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))

(defun gdmir-move-to-relative-point (point-difference)
  (let ((target (+ gd-point point-difference)))
    ;; 80 is a guess at how far we have to be going before moving vertically
    ;; might be of benefit
    (cond ((< point-difference -80)
           ;; Go until you have passed the point.
           (while (> gd-point target)
             (gdmir-move-up) ))
          ((> point-difference 80)
           ;; Go until you have passed the point.
           (while (< gd-point target)
             (gdmir-move-down) )))
    ;; Then we will finish off with side to side movement
    (let ((point-difference (- target gd-point)))
      (if (< point-difference 0)
          (gdmir-move-left (abs point-difference))
          (gdmir-move-right (abs point-difference)) ))))

The motion to the beginning of the line is still annoying but as you can see we sped that up slightly by using the "home" key. This moves the point to the beginning of the line. You don't even have to select in this move as we can calculate the point at the beginning of the line in Emacs. There is one problem with this, however, since lines can wrap, and pressing "home" in Google Docs brings you to the beginning of the visible line, this cannot be used if you are on a wrapped line. I have the program move horizontally until it is within a conservative estimate of the maximum line width, then us the "home" key. You might think it a good idea to use the "shift+home" key combo repeatedly interspersed with "left" key presses counting how far we've gone, but the same issue remains, if we happen to start on some whitespace, we may miss some characters at the end of the line.

Buffering Commands

Buffering the input commands speeds things up in general because it removes some overhead to the input sending process. The way we have it set up right now is to have each keystroke start a bash shell and run the xdotool command to send input to the document. Just starting bash contributes a pretty big overhead. By buffering the commands, we can collect several commands and send them all at once, defeating the overhead. All in all this seems to speed up input by a factor of two, or so. This is a pretty significant improvement³. In fact, this beats the above method of motion for all but very long yet unwrapped lines.

In order to incorporate this kind of buffering from within Emacs, we can just hold of list of pending commands that need to be sent to the document. We will modify the gdmir-send-key function to push onto that list rather than actually send it to via xdotool. If the buffer gets to a certain size, we flush the commands to the document. In addition, since sending a string via xdotool is the last thing one can do in an invocation to xdotool, we also must force a flush of all stored commands before we send the string.

(defvar *pending-keys* nil)

(defun flush-commands (&optional string)
  (let* ((cmd (if *pending-keys*
                  (apply #'concat "key " (mapcar (lambda (x) (concat " " x)) (reverse *pending-keys*)))
                  " " ))
         (cmd (if string
                  (concat cmd " type " string)
                  cmd )))
    (when (or *pending-keys* string)
      (shell-command (concat "DISPLAY=:1 xdotool windowFocus 4194333 " cmd)) )
    (setf *pending-keys* nil)
    (setf *last-flush* (float-time)) ))

(defun gdmir-send-key (keys &optional instant)
  (push keys *pending-keys*)
  (when (or instant (< 20 (length *pending-keys*)))
    (flush-commands) ))

(defun gdmir-send-string (string)
  (flush-commands string) )

In addition to all of this, we will also need to alter several other functions to force flushing as some of them require synchronous execution, like anything that involves the clipboard.

(defun gdmir-move-to-zero-column ()
  (flush-commands)
  (save-excursion
   (goto-char gd-point)
   (gdmir-move-left (max (- (current-column) (- *ems-per-line* 1)) 0))
   (goto-char (- gd-point (max (- (current-column) (- *ems-per-line* 1)) 0)))
   (cond ((< (current-column) *ems-per-line*)
          (gdmir-send-key "Home Right" t)
          (setf gd-point (line-beginning-position)) )
         (t (gdmir-move-left
             (current-column) )))))

(defun gdmir-move-up ()
  (flush-commands)
  (save-excursion
    (goto-char gd-point)
    ;; More to the left most position on the screen
    (gdmir-move-to-zero-column)
    (goto-char gd-point)
    ;; Move to the left of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left")
    ;; move up selecting the differnce
    (let ((orig-line (line-number-at-pos)))
      (loop until (/= (line-number-at-pos) orig-line)
            do (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key shift+Up")
               (let ((selection (gdmir-grab-selection)))
                 (backward-char (length selection)) )
               ;; This doesn't move the point, it just moves to the left of the
               ;; selection and unselects the text.
            (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left") ))
    (setf gd-point (point))
    ;; Move to the right of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))

(defun gdmir-move-down ()
  (flush-commands)
  (save-excursion
    (goto-char gd-point)
    ;; More to the left most position on the screen
    (gdmir-move-to-zero-column)
    (goto-char gd-point)
    ;; Move to the left of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Left")
    ;; move down selecting the differnce
    (let ((orig-line (line-number-at-pos)))
      (loop until (/= (line-number-at-pos) orig-line)
            do (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key shift+Down")
               (let ((selection (gdmir-grab-selection)))
                 (forward-char (length selection)) )
               ;; This doesn't move the point, it just moves to the left of the
               ;; selection and unselects the text.
               (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))
    (setf gd-point (point))
    ;; Move to the right of the code delimiter
    (shell-command "DISPLAY=:1 xdotool windowFocus 4194333 key Right") ))

(defun gdmir-before-edit (start end)
  (gdmir-move-to-relative-point (- end gd-point))
  (setf *prechange-text* (list start end (buffer-substring start end)))
  ;; Delete the string in GD before in Emacs
  (save-excursion
    (goto-char gd-point)
    (loop for i below (- end start)
          do (progn
               (when (= 0 (current-column))
                 ;; Clear out code marker
                 (gdmir-send-key "BackSpace") )
               (decf gd-point)
               (gdmir-send-key "BackSpace")
               (backward-char) ))))

(defun gdmir-after-edit (start end old-length)
  (when (< 0 (- end start))
    (loop for line in (split-for-xdotool (buffer-substring start end))
          do (cond ((eql line 'question)
                    (gdmir-send-key "shift+slash") )
                   ((eql line 'amp)
                    (gdmir-send-key "shift+7") )
                   ((eql line 'ret)
                    (gdmir-send-key "Return")
                    (gdmir-send-string "\\|") )
                   ((< 0 (length line))
                    (gdmir-send-string (gdmir-gdocify-string line)) )))
    (incf gd-point (length (buffer-substring start end))) )
  (flush-commands) )

(defun insert-change-hook ()
  (setf *change-hook-in-effect* t)
  (push
   'gdmir-before-edit
   before-change-functions )
  (push
   'gdmir-after-edit
   after-change-functions ))

And lastly we probably want to set an idle timer to run flush-commands when nothing is happening. This will make it so no changes will sit indefinitely in the key buffer. After waiting around a few seconds (four here) and pending keys are sent to the document.

(setf *idle-flusher*
  (run-with-idle-timer 4 t (lambda () (flush-commands))) )

Two other thoughts of speeding things up come to mind, though I did not attempt this for free time reasons.

1. Asynchonous Input: Buffering commands is also a good idea for a completely
   different reason if we can buffer them outside of Emacs. Emacs, as lovely
   as it is, doesn't allow for multiple threads. This means that we have to
   wait as the commands are processed. If we are able to buffer commands to a
   different process, we could have a concurrent execution. We only need to
   have to synchronize the input with Emacs when we want to get contents of the
   Google Doc, i.e. when we are examining the clipboard.
   
2. Command optimization: The commands can also be optimized. For instance,
   when you yank to an Emacs buffer (a.k.a. paste), Emacs apparently writes it,
   then deletes it, then writes it again. I'm not sure why this happens, nor
   did I ever notice until the mechanism of writing and deleting was slowed
   down by a factor of 1000 or so. In principle, before the buffer is flushed
   to output, it could pass the contents through an optimizer, that would
   reduce this particular delete-redraw cycle to no action.
   

Conclusions

Let me take a moment to relay a few observations I made while developing this interface with a "cloud" program. First, this was very hard and convoluted. I don't think that Google has any interest in stopping me from doing this, but at times I actually thought there was an antagonistic entity on the other end thwarting my attempts. If you copy and paste from a Google Doc, whitespace might get eaten (I think this is happening due to X), or indentation might get messed up (this is almost certainly happening on the Google side). I had weird cases where more than four spaces in a row were converted to tabs, and bizarre rules on what and how much whitespace was eaten at the beginning and end of the selection. You saw how we dealt with this in the code (adding an vertical bar before each line), but it is just the nature of the beast that this is a fragile set up. It would really be nice if Google made a true virtual white-board, or used an existing one (they must exist). They could put a public API on it and anyone trying to do this would be giddy.

Second, because it's a cloud application, Google can upgrade and make incompatible changes at any time. The compatibility doesn't matter much to humans as they are adaptable, but to a program, it really throws a wrench in the works. For instance, I swear that when I first started developing, copying a block of text and pasting it in a Google Doc preserved the indentation. A few days later I tried it and this was no longer the case. It seemed like a bug that would bother people, so I wouldn't be surprised if it is fixed by now. Many people are excited by how fast an application can move when the developer decides the upgrade schedule. I guess I often times prefer the stability of getting to decide that myself.

In the end this worked well enough to use, which is the most important thing. It is pretty basic functionality, but good enough to really help when programming in Google Docs⁴. I wouldn't be surprised at all, though, if I tried this in a few weeks and found that it didn't work anymore. In the very end, however, it was ultimately unused as the phone interview consisted only of pen and paper questions. Not to despair, however, the problem was a fun one to tackle. It included interesting things like pushing the X windowing system further than most users ever do, really exploiting the extensibility of Emacs, and playing with the flexibility of the GNU/Linux operative system in general. A few years ago I would have thought this too hard to actually do and over a decade ago in Windows I would have thought this impossible (this might be possible in OS X, I imagine fiddling with the clipboard and sending keys might be the hard bits). All in all, this was a fun little project during some of the development.

Footnotes:

¹ There is actually a bug here. Emacs sometimes specifies that the
change area before isn't the change area specified in after. For instance,
check the command M-x capitalize-word, which will delete the word, but only
write the first letter of the word. I'm not sure if this is my misunderstanding
of the arguments Emacs sends, or if it is a bug in Emacs itself. Luckily it
seems to be very rare.

² Yeah, I know, if it's selected it is already on the selection clipboard.
I use the ctrl+c/ctrl+v one as it gives a bit more control.

³ To go further, one could attempt to reduce the xdotool delay
between key presses (which defaults to 13 ms). I didn't attempt this as I
assumed that it might reduce the robustness of the method (i.e. keystrokes might
be missed).

⁴ I originally had visions of actually mirroring the buffers.
I.e. you tell Emacs which buffers you want displayed in the document and it will
grab them, insert the code marker, perhaps a simple frame marking the buffer you
are in, and send it to the clipboard to be pasted on the document. I actually
had Emacs sending entire buffers at one point. After I started hitting more and
more issues with clipboards and how Google Docs messes with indentation, or
whitespace, or other things, I decided that this was not worth the effort.