Still not ready for private consumption

Good news! This time, I did manage to find a use for Substrate. It turns out that if you want to suck the motivation out of your programmers and make them want to quit, forcing them to use Substrate is a really good way to achieve that!

So, yeah. Eating your own dog food is a great way to find flaws in your own software, but not necessarily a good way to get them fixed. I mean, I want to fix them, I do, but the idea of eating my own dog food is that I should be using Substrate to manipulate its own source code. So I'm supposed to fix those debilitating flaws using a debilitatingly flawed software. No fun there! It would also take a while, because the sub-par tool would drag me down. Better do it all in vim and forget about the dog thing.


Iteration 3

This iteration is about using Substrate and vim, together. My goal is to make Substrate a good complement for vim.

My goal was never to replace vim. That would be silly! I was not thinking of Substrate as a text editor at all, but as a text manipulator. Sometimes I would need an editor, and sometimes I would need a manipulator. But I have now realized that I won't be developing Substrate using Substrate, I will be developing Substrate using vim! Mostly vim. So if I want to keep Substrate in the loop, to eat it as dog food, then I need Substrate and vim to play along.

And to do that, I need Substrate to interact with the external files I'll be editing in vim.


Story 10 (1 story point): I can save the current project. Use a version number in case the format changes.
Story 11 (1 story point): I can run the script with a particular file as input.
Story 12 (1 story point): I can run the script with each file from a directory (recursively) as input.

Story 13 (1 story point): I can run the script with only the changed files as input.

Story 14 (2 story point): I can perform story 13 from the command line, or when a file change is detected.
Story 15 (1 story point): I can write multiple parallel scripts (all with the same input).
Story 16 (1 story point): I can write multiple sequential scripts (feeding one into the other).

No "bonus" stories this iteration, not anymore. Listing the stories in advance gives me a short term goal to accomplish, and that's motivating. There no motivation in completing optional stories: who am I trying to impress? I just want to be done with the mandatory stories as early as possible, and play video games for the rest of the week.

What one hat said to the other

Developers know which features would be easy to add, but it is users who know which features would be useful to add. This is why, after completing what I thought was the Minimal Viable Product version of Substrate, I decided to trade my Developer hat for a User hat. This turned out to be a very, er, humbling experience.

I wanted to know which lack-of-a-feature was causing the most pain to my non-existent users. To find out, I tried to use Substrate for all the tasks it could handle. And the main conclusion of my experiment was that there were, in fact, very few tasks it could handle. Or, as User hat would put it: What a crappy, buggy, useless software.

Well, thanks for your feedback, User hat. For what it's worth, Developer hat here is more worried about "useless" than about "buggy". That's because I, too, aspire to be a User some day! I want to eat my own dogfood. Substrate is a text manipulation tool, code is made of text... surely one day a version of Substrate will be useful enough for me to use it while developing later versions?

One day, perhaps, but not yet. The current Substrate is great for twea— ok, the current Substrate is a crappy, buggy program aiming to make it easy to tweak long chains of bash commands. But even if it wasn't buggy, the bigger problem is that very few tasks can actually be accomplished through a single long chain of bash commands.


I'm really glad I tried this MVP approach, because this is a major flaw which I could only have discovered by actually using the software, as opposed to merely developing it. I have long paid lip service to The Unix Way, and I am not alone in thinking that a few carefully chained Unix commands can go a long way. But not, it would seem, all the way.

Since my long term vision for Substrate was a GUI for creating complex chains of text-manipulating nodes, I need to revise my approach. And that's precisely what short iterations are for, aren't they?

Iteration 2

In this iteration, I will focus on adding new features, not on fixing the many bugs User hat found. Hopefully, by the end of the iteration, Substrate will evolve into a crappy, buggy, marginally useful product. Useful enough, I hope, for Developer hat to use.

Story 5 (1 story point): I can enter different inputs for the script.
Story 6 (1 story point): I can run the script on a series of different inputs.
Story 7 (2 story point): I can save the script and its inputs as a single project.
---
Story 8 (2 story point): I can write multiple scripts which call each other.
Story 9 (2 story point): I can write a script which applies multiple inputs to another script.


Adjusting from last iteration, this time I intend to finish the first three stories in two weeks, not one month. I have also added two optional stories, in case I finish early again. This is my personal extension to the Agile process: even if your velocity indicates that you can only deliver 4 story points per iteration, you should still schedule a bit more than that. I'll let you know if it's a great way to empirically measure by how much to increase your velocity, or just another idea which sounds better to one hat than to the other.

Success!

Has it only been a week? Wow. This challenge was both easier and harder than expected.

Easier, because I expected to finish in a month, rather than a week. I'm really bad at making time estimates.

Harder, because I ended up writing much more code than I expected. I initially thought that I would implement the four stories back-to-back, with a few lines of code each. That was the plan. But I ended up organizing my code a lot, and implementing a few visual niceties which were not in the original stories. I'm really bad at sticking to a plan.


In any case, tada! Here is Substrate, iteration 1.

The greyed-out text is one of the extra visual niceties
I couldn't help but add. This shows you which lines
are going to run when you press F5.

That's right! I escaped from my curse, and completed a project: an MVP version of Substrate. I think laying down a plan at the beginning really helped, even if I didn't follow it too closely. I'll definitely need to write down new stories for iteration 2. But not yet.

For now, I need to play a bit with the product. Only then, as a user, will I have the authority to decide which parts of the product could be improved.

May the day I actually complete something finally come

Completing projects is hard

I have a curse.

My curse is that I have too many great ideas. Now, I know that ideas are cheap, so I don't just publicly announce my ideas and then complain that nobody is acting on them. I know that if I want my ideas to have an impact, I must act on them myself. I do act! But I'm still cursed.

Often, my idea is something that I can code on a computer. That's great! I have a computer, and I know how to code. I can act. So I put everything aside, fire up my favorite text editor, and enthusiastically put in the hours necessary to create something out of nothing. But it doesn't work! I'm still cursed.

The main problem with this "put everything aside" approach is that the main thing I put aside is usually the code I was writing in order to implement my previous idea. And by the time the code for my new idea becomes something usable, it will in turn be shoved aside by an even newer, better idea! That's why great ideas are my curse. If only I was coming up with worse and worse ideas instead of better and better ones!


Well, this curse ends now.

I was discussing Agile with Nadya yesterday, and I asked her how small she thought a team could be while still being Agile and benefiting from it. Since she and I love to work on common projects, I expected her to answer "two", suggesting that our common projects should use Agile. But to my surprise, she answered "one".

I guess it makes sense. If following the Agile methodology can help a team to ship, why couldn't it help me complete my personal projects? Let's give it a shot.


The project I will actually complete

My current project is Substrate, a text processing tool making it easy to build text processing pipelines.

I got that idea while using Houdini, a 3D software package making it easy to build 3D-model and 2D-image processing pipelines. It's very different from other 3D packages. Using Houdini was much more fun than using 3DS Max. Houdini felt more like programming than modelling, but at the same time, it also felt more fun than programming. Very few things do! I suddenly wanted all of my tools to be like Houdini.

I started immediately, and that was my first mistake. I should have taken the time to figure out what it was that made Houdini so different. Instead, I immediately started working on a Houdini clone, with the major exception that my clone would manipulate text, not 3D models.

So far, I have a pane system where I can open and close tabs, split panes horizontally and vertically, and... well, that's it, really. No text manipulation yet. I just copied Houdini's pane system because Houdini had it. From an Agile perspective, the state of the project is highly embarrassing: totally unshippable, with the "team" focussing on miscellaneous GUI features over the core functionality.


Well, this ends now.

I still don't know what it is that makes Houdini feel different, but I will find it. Iteration after iteration. And each iteration will be shippable. Amen.


Iteration 1

Since I am working on this during my free time, I will need relatively long iterations in order to have enough coding hours to produce anything. Let's say one month: I shall have a shippable MVP at the end of February. And here is what this MVP will contain.

Story 1 (1 story point): I can load a Bash script and view its contents.
Story 2 (1 story point): I can run the currently-loaded script and run it using a command from the menu. I will see the script's output on stdout.
Story 3 (1 story point): I can run a prefix of the script using a command from the menu. I will see the output of the truncated script on stdout, or an error if the truncated code doesn't represent a valid Bash script.
Story 4 (1 story point): I can successfully run a script prefix even if the last line ends with a pipe character.

The hypothesis tested by this first iteration is that while working on a pipeline, the ability to see intermediate results from the middle of the pipeline will make it much easier to debug the pipeline. The Houdini feature which corresponds to this is the ability to click on any node to see what the 3D model looks like at this point.

I'm not convinced that this is really what makes Houdini fun to use, but it's certainly a core feature, and it's one which I am confident I can implement quickly.


Well, let's get to work!