HtDP for the experienced programmer

I am a fan of Scheme (a programming language for the soul) and SICP "the greatest book on programming ever written", but my hat goes off to the creators of DrScheme, a great interactive environment for the beginner and experienced programmer alike.

The other night I swallowed my pride and started to work through HtDP (How to Design Programs), the PLT Scheme gang's reaction to the fact that SICP (which came from MIT) while great for the suitably talented individual does not by-and-large work for the mainstream. In this book they explicitly teach program design, a systematic approach to thinking and problem-solving that it seems only a few people get by the more traditional osmotic process.

HtDP is covers a great range. Although principally geared to college level, it can be taught in high schools (at a slower pace), and can teach most experienced practitioners more than a thing or two.

Additionally, the people on the PLT Scheme mailing list are incredibly friendly and supportive. Here's what Matthias Felleisen, first author of HtDP (among many other things) wrote when I mentioned that I was starting to work through HtDP, but was having difficulty restraining myself from skipping ahead:

If you are an experienced programmer, you should read HtDP like this:

• Read the sections whose title starts with "Designing ...."
• Also read the "iterative refinement" sections
• Pick five exercises in the preceding and/or follow-up section and solve them according to the recipe
  [my emphasis]
• Unless you're stuck move forward
• Try to understand the "symmetry" between data definitions and templates

I expect that somewhere around late part II, you will slow down. You may pick up real reading as of part III, though some may make it thru III and only "stutter" in IV.

Also:

• Use check-expect to express your examples/tests
• Avoid draw.ss exercises, replace them with world.ss but that's a non-trivial switch

And holler if you are having trouble -- Matthias

In short: Great tools, great teaching materials, supportive and experienced community.

Inspiration

Bob Franklin, who programmed up the first electronic spreadsheet, VisiCalc, in 1978 working with Dan Bricklin -- whose idea it was -- has been writing up the history. I found the following inspirational and salutory. How to build a software product:

Design Principles

VisiCalc was a product, not a program. Decisions were made with the product in mind and, to the extent possible the programming was towards this end. In practice it was more complicated as we were designing against the limitations of the personal computers, price point and, most important, what the user could understand.

The goal was to give the user a conceptual model which was unsurprising -- it was called the principle of least surprise. We were illusionists synthesizing an experience. Our model was the spreadsheet -- a simple paper grid that would be laid out on a table. The paper grid provided an organizing metaphor for a working with series of numbers. While the spreadsheet is organized we also had the back-of-envelope model which treated any surface as a scratch pad for working out ideas. Since we were used to working with powerful computers without worry about the clock running, we already had the experience of focusing on the users needs rather than the computers needs.

The ability for Dan and I to work as a team was crucial. While he could've written the program, the fact that he wasn't gave him the freedom to focus on what the program should do rather than how to do it. I could appreciate his reasons and would eventually accept that I had to change code that I had labored over. We were able to find ways to take advantage of the limited space available for the program in deciding what features to include or not include.

The original version put the entry area at the bottom of the screen. By playing with this simple prototype Dan found that it was better to put the entry area at the top of the screen and I made the change to the evolving program.

In addition to prototyping, Dan put together a reference card for users. If we couldn't figure out how to explain a feature on the reference card we would change the program. The original method for copying formulas was too complicated so we just changed the design rather than try to explain it.

The more things change, the more they stay the same.

Alan Kay's Reading List

Alan "Smalltalk" Kay gives a list of books for people interested in the ideas and philosophies that influenced Squeak. It looks like a good list for anyone interested in software invention.

I have read a smattering of the books and some other works by authors on the list, and they were generally outstanding. I will try to sample the remainder (over time).

Programming Yin and Programming Yang

Q: Why is programming fun?
A: It is a form of play and exploration in which a "one-time" "small" effort (writing the program) yields a large reward (the program does something). It combines the artistic joy of creation with the scientific reward of nutting out a puzzle.

Q: Why is programming hell?
A1: When the program fails to produce the expected results a diagnostic process of "debugging" follows. As the program grows in size and power and (hence) complexity, this debugging phase dominates the programmer's time, and brings mainly relief rather than reward.

A2: When requirements change the program may need to be re-jigged to accommodate them, and this re-jigging, similar to debugging lacks the immediate rewards.

* * *

So, to increase the rewards of programming -- and incidentally productivity and profitability -- techniques are sought to reduce the dross (debugging and re-jiggering), and increase the rewards (features and elegance).

On the Yang side of programming I include design, architecture, algorithm invention, and implementation of new features.

On the Yin side I include practices like Design By Contract (DBC), Test Driven Development (TDD), and Continuous Integration (CI). These provide no up-front reward, but pay for themselves over time.

Interestingly, all three involve using programming to write better programs. DBC and TDD may be regarded -- with some licence -- as simple forms of meta-programming. We are writing program fragments to help test our programs. Also, the act of making tacit assumptions explicit helps us to think about and hence improve the underlying design.

By putting in place these effective (Yin) measures, we can be bolder (Yang) in adding new functionality, having built a software safety net as we step forward.

True meta-programming, writing programs that write programs brings us back to the Yang side.
We are now truly working at a higher level of abstraction, with correspondingly more leverage, but when bugs appear in our meta-programs the debugging gets harder too ...

Bit rot vs Re-usable code

The intellectual property in a software company is diffuse. The explicit IP can be found in patents, papers, documents and, above all, the code base. The tacit IP exists in people's heads.

It is the tacit IP that allows for innovation and change. Indeed, a good way of understanding the possibility of change is the extent to which the maintainers of the software have a theory of how it works. If they do, they can change and extend it. If they cannot they are likely damaging the conceptual integrity of the code-base.

In The Worthlessness of Code James Turner advocates regularly rewriting to ensure that the software development team understands what they are working with, and in-particular, so that the original designers effectively refresh and transfer their knowledge to new hires.

In the comments the rebuttals of "continuous re-writing", and "re-usable code" are raised. Personally, I sympathize with Turner. I used to have a motto: Before re-use, use. This mocked the scenario of people trying to writing libraries or frameworks that would be used multiply, before being deployed successfully even once. Oy! The better alternative is to extract, abstract and generalize after an initial success or two.

What about libraries? The good ones do have a long life. This comment from the Reddit comments on Turner's article draws an important distinction:

How about "reusable code is an asset, single-use code is a liability"?

I like this idea. There is the abstracted, cleaned stuff that ends up in libraries, and gets re-used, and there is the more scripty stuff that glues things together and gets the short-term job done.

This raises the possibility of a new, somewhat informal metric: Ratio of multiple-use (quality code) to single-use (get-it-done code).

Do you want macros with that?

From an introduction to Scheme by Ken Hickey (my emphasis):

Just as functions are semantic abstractions over operations, macros are textual abstractions over syntax. Managing complex software systems frequently requires designing specialized "languages" to focus on areas of interest and hide superfluous details. Macros have the advantage of expanding the syntax of the base language without making the native compiler more complex or introducing runtime penalties.

Beautifully expressed. I say, "just give me the power, already", which most programming languages -- with honorable exceptions -- withhold.

Probably not everyone should be playing with this kind of power, but while I do not allow my 3-year-old son light the candles, I hope that one day he will have the maturity and coordination
to do the job. At that time I assume that matches will be available.

Programming is Theory-Building

I was exposed to the notion that the activity of programming is intimately tied up with a hidden process of constructing a theory in an excellent Google Video talk by Peter Seibel, author of Practical Common Lisp. It appears that this notion has its origin with Peter Naur -- one half of the "Backus-Naur form" -- in his article Programming as Theory Building. Naur writes:

[P]rogramming properly should be regarded as an activity by which the programmers form or achieve a certain kind of insight, a theory, of the matters at hand. This suggestion is in contrast to what appears to be a more common notion, that programming should be regarded as a production of a program and certain other texts.

This insight has ramifications for the anthropology and life-cycle of large projects, where in order to produce modifications to an existing program without compromising the structure the programmers must understand the underlying theory. Since such an understanding is not easily acquired, we have Fred Brooks' observation that adding more manpower to a late project makes it later as a corollary: The people who understand the theory have to induct the newcomers, thus soaking up both groups' productivity.

Having had the experience myself of being brought late onto an existing project, I can appreciate first-hand how hard "learning the theory" can be. At other times I have developed sophisticated algorithms that I don't think anyone else who came after was game to touch!

Understanding and acknowledging this element of software development should be helpful in managing large-scale and long-term developments. Of course, since it accepts the human side of programming, no simple "solutions" follow, but its implications for quality assurance, recruiting, and -- as Naur points out -- the status of programming as a profession, are far-reaching.