Sunday, May 20, 2012

Thank you, KeyRemap4MacBook!

At long last! I managed to fully reproduce my Linux xmodmap layout on OS X.


I don't think I have particularly peculiar typing needs. I often type code, which frequently uses characters like "[", "{", and "<". Those are easy to type using the US keyboard layout. I also often type French, which frequently uses accented characters like "é", "à", and "û". In that situation, most people opt for switching between keyboard layouts depending on the situation. I, however, opted for a custom keyboard layout.

My custom keyboard layout is mostly the US keyboard layout, plus some Alt+key combinations for dead keys like "´", "`", and "^". On Linux, this was easy to setup using xmodmap. Implementing the keyboard layout on OS X was a bit more complicated, because its XML representation of keyboard layouts is very long and it's not easy to test changes incrementally, but I eventually managed to do it.


Then one day, I decided that the modifier keys were too low.

I am not even using Emacs, but typing Ctrl with my pinky is really twisting my hand in a way I don't like. So I bought an ergonomic keyboard. It didn't help.


One of the reasons I chose this particular keyboard was because of the two little extra arrow keys below the space bar. I thought I could remap them to Ctrl and Shift, thereby allowing me to type those troublesome modifiers using my thumb instead of my pinky. That didn't work either. The keyboard preferences which were installed with the keyboard helpfully allow me to remap those to any other key... except modifiers. No luck directly modifying the layout files either, not even through keyboard layout creation tools like Ukulele. I did manage to turn Caps Lock into an extra Ctrl, but the other modifiers looked like they would be stuck in place forever.


I had no such problems with Linux. Again using xmodmap, I converted the key above the enter into an extra Shift key, much to the relief of my right pinky. The only annoyance with this solution was that I kept trying to use that key as a Shift on the Mac computer too, only to be presented with a disappointing backslash character as a result.

Well, those days are now behind me! Today, I downloaded KeyRemap4MacBook, a very helpful tool which managed to accomplish what so many others had failed. I am now typing this text on my Mac, using the backslash key as a Shift! And my right pinky is very grateful.


Thanks, KeyRemap4MacBook!

Sunday, April 29, 2012

Building qtHaskell on OS X


Finally! I spent the entire weekend on this, but I finally managed to compile qtHaskell on OS X. I spent a lot of that time on the internet googling for the error messages I encountered, to no avail. So, to save some time to the next person who google them, here is what worked for me.


The qtHaskell documentation recommends that you use their ./build perl script. That almost works, but not quite. Thankfully, by giving arguments to the scripts, we can run the parts which work, and skip over the parts which don't.

Speaking of arguments, one the most annoying errors is the "scattered reloc r_address too large" error message which only occurs at the very end of a very length compilation. I stumbled upon a page recommending to use "--extra-ld-opt=--architecture=x86_64" to fix a similar error message, and since then I have superstitiously stuck the argument everywhere I could, just to be sure.


So, let's build qtHaskell, shall we? First of all, qtHaskell is a set of Haskell bindings for Qt, so you need to install Haskell and Qt. I used Qt 4.7.1, qtHaskell 1.1.4, and GHC 7.0.4 (64 bits). Actually I had the 32 bits version of GHC at first, and that let to the aforementioned r_address problems, followed by linking problems afterwards. Better upgrade to 64 bits!

brew uninstall ghc
brew install ghc --64bits

We should now be ready to compile qtHaskell proper. There are actually two parallel hierarchies to be compiled, the C++ implementation and its Haskell FFI interface. Both are very large, because, well, Qt is large.

First, the C++ part: no problems here, the build script works just fine for this part.

./build user cpp qmake cmake \
    --extra-ld-opt=--architecture=x86_64
./build user cpp-install \
    --extra-ld-opt=--architecture=x86_64 --no-sudo

Note that even though the "user" flag was given, the build script will still
install the files globally in /usr/local/lib/libqtc_*. Presumably, the flag is only for the Haskell part, for which the steps would ideally be as follows.

./build user haskell configure \
    --extra-ld-opt=--architecture=x86_64
./build user haskell build \
    --extra-ld-opt=--architecture=x86_64
./build user haskell-install \
    --extra-ld-opt=--architecture=x86_64 --no-sudo

Unfortunately, the middle step fails with the following error message.

unrecognised command: makefile (try --help)
runghc Setup.hs makefile: No such file or directory

Fear not! This cryptic-looking error message is actually a clue towards the solution. Apparently, "runghc Setup.hs makefile" is the command used for building the Haskell part. Using the help, as recommended in the error message, leads to the following alternative build commands.

runghc Setup.hs configure --user \
    --extra-ld-opt=--architecture=x86_64
runghc Setup.hs build

Unfortunately, that doesn't work either, at least not with a recent GHC. There is a problem in the qtHaskell code, which leads to the following error message.
Qtc/Core/Attributes.hs:583:13:
    Could not deduce (Qstt a (QDialogSc b))
      arising from a use of `slotReject''
    from the context (Qstt a (QDialogSc b1))
      bound by the instance declaration
      at Qtc/Core/Attributes.hs:581:10-52
    Possible fix:
      add (Qstt a (QDialogSc b)) to the context of
        the instance declaration
      or add an instance declaration for (Qstt a (QDialogSc b))
    In the expression: slotReject'
    In an equation for `reject'': reject' = slotReject'
    In the instance declaration for `QsaSlotReject a'


If you are familiar with Haskell, you should be able to fix the problem on your own. Alternatively, you could grab Uduki's hsQt fork of qtHaskell, which comes pre-patched. Or you might apply the following patch yourself:
diff --git a/Qtc/Core/Attributes.hs b/Qtc/Core/Attributes.hs
index 197c506..217d585 100755
--- a/Qtc/Core/Attributes.hs
+++ b/Qtc/Core/Attributes.hs
@@ -580,7 +580,7 @@ class QsaSlotReject w where
 
 instance (Qstt a (QDialogSc b)) => QsaSlotReject (a) where
   slotReject' = (Qslot "reject()", \_ -> ())
-  reject' = slotReject'
+  reject'     = (Qslot "reject()", \_ -> ())
 
 class QsaSignalRejected_nt_f w x f where
   signalRejected', rejected' :: x -> SltConf w f

Now that the source is patched, the Haskell part should finally compile.

runghc Setup.hs build

If you run out of memory during this phase, just run the command again. And again. As many times as it takes to compile all the targets. When you get the the very last target, qtHaskell is finally linked... and then, like me, you might get one of those nasty "scattered reloc r_address too large" errors. Did you? If so, you might have skipped the very first step and be stuck with a 32 bits version of GHC. Check using "ghc --info".

If you insist on using the 32 bits version, you could try the following variant of the build command, which got me through that step. Still got troubles when linking programs using qtHaskell, though.
runghc Setup.hs configure --user --disable-library-for-ghci

Now that the Haskell part is built, we can finally install it! This time, the "user" flag is honored.
./build user haskell-install \
    --extra-ld-opt=--architecture=x86_64 --no-sudo

Let's test this! The qtHaskell helpfully provides the following Hello World code, just put it in a file with the "hs" extension and build it into an executable using "ghc --make".

module Main where

import Qtc.Classes.Qccs
import Qtc.Classes.Gui
import Qtc.Gui.Base
import Qtc.Gui.QApplication
import Qtc.Gui.QPushButton

main :: IO Int
main = do
  qApplication ()
  hello <- qPushButton "Hello qtHaskell World"
  resize hello (200::Int, 60::Int)
  qshow hello ()
  qApplicationExec ()
Tada!
At long last. A button!

Saturday, March 31, 2012

As the story dragged on


The main reason I decided to experiment with non user-visible stories was to keep stories short. That part worked great; as long as they were short, I breezed through the stories and was eager to start the next. But then one particular story — I'm looking at you, story 19! — turned out to be a lot more work than expected. As the story dragged on, I lost my motivation... and then it took even longer to get it done. I would like to avoid this situation in the future, if possible.

But how? By adjusting my process, of course! Ideally, I should be able to complete each story in one sitting; and now that stories don't need to be user-centric, there is no reason not to make them as small as needed. If they aren't, then the plan needs to be refined. Thus, this time, I am going to break something even more fundamental than the user-centric nature of stories: the fixed-length iterations! How naughty.

This iteration will end in one week, or when a story drags on for more than a day, whichever happens first. If it is the latter, I will have the opportunity to adjust the offending task, by dividing it into smaller chunks.


Alternatively, I could rethink my design so that the offending task is not needed, or vastly simpler. Or just different. Or more fun to code.

I mention this possibility because I stumbled upon a post from a writer explaining how she quintupled her productivity by optimizing her process, and she found out that she was more productive when writing scenes she was enthusiastic about. Well, I was enthusiastic about Story 20. And hated Story 19. I'm not sure why.

In order to figure it out, from now on I am also going to write down the best and the worst story of each iteration. Hopefully, a pattern will emerge. Although, to be honest, even if I had an easy criterion to distinguish stories which are going to be fun from those which are going to be boring, I don't really know what I could do with that information. Sure, some stories are less pleasant than others, but they still need to get done, right? Unlike passages in a book, if the code is boring to write, it doesn't mean the user won't enjoy the result. Who knows, maybe I'll find a different software organization which minimizes annoying tasks?


Iteration 5

As yet another crazy departure from conventions, I'm dropping story numbers. They haven't been very useful so far.
  • Setup continuous compilation (using Substrate to detect when the source files change).
  • Use signals to propagate errors.
  • Implement an InputNode which can bake itself.
  • Implement a Gui which links the input nodes with the input pane widget.

Saturday, March 17, 2012

Who cares about the user?

As promised, I am going to play with my process. Unlike what I promised, I am not going to waste an entire iteration on this. I am going to waste several!

I have already tried one change: scheduling optional stories. It didn't work, but I'm glad I tried. I am not on a tight schedule, so I can take the time to experiment and see what works. I even tried the Spiral idea of trashing all my code and starting from scratch using my newly accumulated experience: that's what I did on the very first iteration, when I decided to drop everything and start fresh, Agile-style this time. What should I try for the next iteration?


Well, is there something from the last iteration which could be improved? Let's see, was there anything at all which went wrong? Let me think. Oh, right. Maybe the fact that I totally failed to produce anything at all?

Yeah, during the past iteration, I didn't accomplish much. Or rather, that's what you would think if you only looked at the number of stories I completed: zero. And indeed, when I run Substrate, it looks exactly as it did two weeks ago. But that is only part of the picture! Sure, the look didn't change and there are no new buttons to click on, but I did spend a lot of time working on Substrate, trying out new architectures and comparing design approaches. Even as I worked, I felt like I was wasting precious time, because days went by and my metric wasn't improving.

Well, screw the metric. Screw the users, I mean. Erh... that's not what I meant either.


This iteration, I am testing a variant on story descriptions: instead of focusing on user-visible features, I am going to focus on developer-visible features. If code needs to be written, then I don't care whether that code is producing buttons, menus, bells, or whistles: by golly, this code is going to be divided into small story-like chunks and assigned to an iteration!


Iteration 4

This iteration is still about geting Substrate to interact with external files, but the steps to reach this goal are more detailed. Stories 18 and 20 have no user-visible impact.

Story 10 (1 story point): I can save the current project. Use a version number in case the format changes.
Stories 11 to 16: Postponed.
Story 17 (1 story point): I can reload and run the entire project.
Story 18 (1 story point): Simple filesystem-backed key-value store with one file per key.
Story 19 (2 story points): The project reloads when the filesystem copy of the store changes.
Story 20 (2 story points): Specialized store interface for strings, integers, and string arrays.
Story 21 (1 story point): The project stores the active input tab.


p.s.: Don't take the title of this post seriously. The only user Substrate ever had, so far, is myself.

Thursday, March 15, 2012

Unifying the Agile and Spiral models

If, like me, you have never heard of the Spiral model of development, here is a summary: it is a bit like Agile, except that all the code is thrown away at the end of the iteration. Also, the scope increases each time, so the iterations grow longer and longer.

Yes, I know this sounds like a terrible idea, but that's only when compared to Agile as a reference point. From that angle, Waterfall is the model where you only get one, very big, very long iteration, and hope to get it right the first time. When put in the context in which it was invented, doesn't the Spiral idea of starting with a smaller version of the project seem like a great improvement?

Not throwing away the code all the time sounds like an even better improvement, but when I first heard of the Spiral model (just a few hours ago), I thought the opposite. Throwing code away! What a novel and intriguing idea! Should I try? Maybe? Being in an Agile team doesn't guarantee good design, after all. And changing a bad design is sometimes a large endeavor. With Substrate, for example, I haven't been able to implement any story at all this iteration, because I was busy refactoring the entire application to use MVC.


Instead of being forced to either drop all the existing code or keep all of it, I suggest we should be allowed to do either. And those two possibilities are just two ends of a continuum: using version control, we can easily revert to a number of intermediate steps between the two extremes, whichever is most convenient.


An Example

Let's start with an empty project, in the middle of Design Space, the space of all possible designs.



We discuss the product idea, draw a bit of UML, and end up with a particular design (a particular point in Design Space).



A few iteration go by, moving the project toward the goal established in the design.



While implementing new features, suppose we come up with a new design for the project. Could happen! Otherwise, we would still be doing big design up front and it would work great. The goal is moved a bit to the left.



Since we are using iterations, we adapt to this unexpected change without a sweat. We just start writing the new code with the new code in mind!




But what if the new design was very, very different from the first?



In that case, starting from scratch is less work.



But this is not an all or nothing decision. Yes, you can start from scratch, but you can also start from any other past point. Just pick the one which represents the least amount of work!



Future Work

Now, as I said, I only heard about the Spiral model today, so there are still many rough corners in my generalization of it and Agile. For one thing, going back in time means removing some features, and if those features have shipped already, maybe you don't want to remove them. Feature Space is not necessarily aligned with Design Space.

The other big problem is that history is linear, while Design Space is multidimensional. When your goal changes its position, it probably won't move across all the dimensions, only some of them. So while it is true that some of your code can be thrown away, most of it probably shouldn't; in order to reach the commit you want to restart from, you will have to throw away some commits which are still good, just because they were submitted after the restarting commit. I recommend using a history-rewriting tool such as git, but when I tried, solving all the merge conflicts was still a lot of work.


Ideally, each new feature could be implemented on a bare-bones version of the application, containing only the elements on which the new feature builds. In that setting, implementing the new feature is much easier, because there are no unexpected interactions with the other features, the codebase is small enough to be understood all at once, and the debug-edit-compile cycle is fast. Theoretically, using feature branches, maybe this ideal could even be reached!

The problem, of course, is that the unexpected interactions bite back once we try to merge all of the feature branches into a complete product.

I think I am going to try anyway. If I can afford wasting an entire iteration redesigning my code, maybe I can also waste one redesigning my process?

Tuesday, February 28, 2012

Still not ready for private consumption


Good news! This time, I did manage to find a use for Substrate. It turns out that if you want to suck the motivation out of your programmers and make them want to quit, forcing them to use Substrate is a really good way to achieve that!

So, yeah. Eating your own dog food is a great way to find flaws in your own software, but not necessarily a good way to get them fixed. I mean, I want to fix them, I do, but the idea of eating my own dog food is that I should be using Substrate to manipulate its own source code. So I'm supposed to fix those debilitating flaws using a debilitatingly flawed software. No fun there! It would also take a while, because the sub-par tool would drag me down. Better do it all in vim and forget about the dog thing.


Iteration 3

This iteration is about using Substrate and vim, together. My goal is to make Substrate a good complement for vim.

My goal was never to replace vim. That would be silly! I was not thinking of Substrate as a text editor at all, but as a text manipulator. Sometimes I would need an editor, and sometimes I would need a manipulator. But I have now realized that I won't be developing Substrate using Substrate, I will be developing Substrate using vim! Mostly vim. So if I want to keep Substrate in the loop, to eat it as dog food, then I need Substrate and vim to play along.

And to do that, I need Substrate to interact with the external files I'll be editing in vim.


Story 10 (1 story point): I can save the current project. Use a version number in case the format changes.
Story 11 (1 story point): I can run the script with a particular file as input.
Story 12 (1 story point): I can run the script with each file from a directory (recursively) as input.
Story 13 (1 story point): I can run the script with only the changed files as input.
Story 14 (2 story point): I can perform story 13 from the command line, or when a file change is detected.
Story 15 (1 story point): I can write multiple parallel scripts (all with the same input).
Story 16 (1 story point): I can write multiple sequential scripts (feeding one into the other).


No "bonus" stories this iteration, not anymore. Listing the stories in advance gives me a short term goal to accomplish, and that's motivating. There no motivation in completing optional stories: who am I trying to impress? I just want to be done with the mandatory stories as early as possible, and play video games for the rest of the week.

Saturday, February 18, 2012

What one hat said to the other

Developers know which features would be easy to add, but it is users who know which features would be useful to add. This is why, after completing what I thought was the Minimal Viable Product version of Substrate, I decided to trade my Developer hat for a User hat. This turned out to be a very, er, humbling experience.

I wanted to know which lack-of-a-feature was causing the most pain to my non-existent users. To find out, I tried to use Substrate for all the tasks it could handle. And the main conclusion of my experiment was that there were, in fact, very few tasks it could handle. Or, as User hat would put it: What a crappy, buggy, useless software.


Well, thanks for your feedback, User hat. For what it's worth, Developer hat here is more worried about "useless" than about "buggy". That's because I, too, aspire to be a User some day! I want to eat my own dogfood. Substrate is a text manipulation tool, code is made of text... surely one day a version of Substrate will be useful enough for me to use it while developing later versions?

One day, perhaps, but not yet. The current Substrate is great for twea— ok, the current Substrate is a crappy, buggy program aiming to make it easy to tweak long chains of bash commands. But even if it wasn't buggy, the bigger problem is that very few tasks can actually be accomplished through a single long chain of bash commands.


I'm really glad I tried this MVP approach, because this is a major flaw which I could only have discovered by actually using the software, as opposed to merely developing it. I have long paid lip service to The Unix Way, and I am not alone in thinking that a few carefully chained Unix commands can go a long way. But not, it would seem, all the way.

Since my long term vision for Substrate was a GUI for creating complex chains of text-manipulating nodes, I need to revise my approach. And that's precisely what short iterations are for, aren't they?


Iteration 2

In this iteration, I will focus on adding new features, not on fixing the many bugs User hat found. Hopefully, by the end of the iteration, Substrate will evolve into a crappy, buggy, marginally useful product. Useful enough, I hope, for Developer hat to use.

Story 5 (1 story point): I can enter different inputs for the script.
Story 6 (1 story point): I can run the script on a series of different inputs.
Story 7 (2 story point): I can save the script and its inputs as a single project.
---
Story 8 (2 story point): I can write multiple scripts which call each other.
Story 9 (2 story point): I can write a script which applies multiple inputs to another script.


Adjusting from last iteration, this time I intend to finish the first three stories in two weeks, not one month. I have also added two optional stories, in case I finish early again. This is my personal extension to the Agile process: even if your velocity indicates that you can only deliver 4 story points per iteration, you should still schedule a bit more than that. I'll let you know if it's a great way to empirically measure by how much to increase your velocity, or just another idea which sounds better to one hat than to the other.