Lambda the Ultimate

The answer: local state is nice; global state isn't.

Regarding your larger issue, it looks to me that dynamic scoping of immutable values and declarative concurrency can coexist without problems. It's only with shared mutable variables that dynamic scoping becomes problematic.

Can you elaborate on this solution? How would invoking an operation in IE1 cause the IE2 counter to be incremented? Remember that the interface IE1 must not change at all. I don't believe you can do it, but I am willing to be convinced otherwise.

I am a Haskell programmer, and I agree there is no easy way to solve this problem, and frankly I think it's a failing of Haskell though not the purely functional paradigm, if you interpret it broadly enough. There is a neat solution in a more general language which avoids the pitfalls of global state and retains the advantages of what you call declarativeness, namely your second alternative:

Or you decide to hide the monadic operations from the programmer and let the compiler add them. In which case you have just implemented a stateful language on top of Haskell.

I think the way you write this it sounds like a sort of compiler hack or blue-sky solution, but it can be given a formal and practical account thus: Haskell, being a purely functional language, can be assigned a trivial monadic semantics, namely over the identity monad. To solve the instrumentation problem, interpret the modules to be instrumented over a state monad. This would require adding another module in some extended language Haskell' which, when you want to run the program as normal, interprets programs in the identity monad, and when you want to run them instrumented, interprets them in the state monad. In other words, this extended language has a feature which lets you run arbitrary code in some particular monad.

This is, really, almost the same as your first solution, namely to make all the code monadic---that is, rewrite it all using do-syntax or >> and return---except that the compiler takes care of the work.

The advantage of this versus using a conventional procedural language with global state is that you retain the ability to easily reason about the program as it will normally be used, in the identity monad, but can still instrument it modularly; indeed, by picking other monads for the semantic interpretation you could conceivably create other interesting variations; for example, if the program is a search program, you could change it's behavior from depth-first to breadth-first, or alter it to produce all results rather than the only first-encountered result. Maybe it is even the case that there is some theorem about monad transformers which lets you easily transfer theorems from the uninstrumented to the instrumented case.

I hope that was intelligible. I can make it more precise, if you like.

Just a random idea!

How different is this kind of reasoning from what you would do with a well-designed imperative system? I suspect not much different.

Can you design equally bad systems, and equally good systems, with either approach? I suspect so.

I'm guessing. Can anyone not guess and still provide answers?

One thing is certain. Bad designers create bad designs in any language.

module E1 ... operation:: T

module E2 ... operation:: (?callback :: ()) => T

It wouldn't be the IE2 counter, it would be a counter in A incremented whenever a callback function supplied from A to E2 was invoked. You're right that the interface IE1 would have to change, but with implicit parameters that wouldn't necessarily imply any change in B, C, or D, and that was the main requirement. Unfortunately, there is nothing like "implicit results" in the Haskell extension proposal I linked to, so the callback function would have to use unsafePerformIO to increment the counter in A.

I'd also like to point out that adding a plain counter to E is just bad design. What if the next step of instrumentation required tracking the time spent in E's operations, instead of number of their invocations? Would you recommend creating interface IE3 to do that? It makes more sense to create a single interface IE whose every operation invokes callback functions StartEvent and EndEvent, provided as parameters to the module.

Of course, standard Haskell doesn't provide anything approaching parameterizable modules nor implicit parameters. But that's the fault of Haskell, not the functional programming in general. Any of the two features would be a better solution to the stated problem than a hidden state.

My point was that dynamic scoping is an obvious feature of choice when you need something that 'lets information pass "underground" between two interfaces'. There certainly are examples of problems where private state would be preferrable to anything else, but this is not one of them.

You finesse the situation by calling unsafePerformIO, but this is using true state so it doesn't count. So I still don't believe you can do it, unless you can provide me with a more complete solution.

How different is this kind of reasoning from what you would do with a well-designed imperative system? I suspect not much different.

This is the point. It is the overall simplicity of the system that counts, not whether it is written in a functional or stateful style. Using true state in the right places makes the system simpler overall.

How about scalability? I can see that some state does not harm but indeed makes things simpler. But does this approach scale? What if you need more than just one counter. Of course, you don't find out about this requirement ahead of time but while the program evolves.

Well, you can always limit yourself to a single paradigm (e.g., functional or OOP) and your results will be as scalable as that paradigm. Using more than one paradigm can only do better.

Perhaps a more important question is, are multiple paradigms needed in a single program?. I think the answer to this question is yes, and that this is obvious when you look at any program bigger than a toy example. For example, you might want to implement an event manager (see section 7.8.5. in CTM). It takes any number of event handlers, where each handler is a pure function from Event x State -> State. The event manager handles both the concurrency of having more than one event handler and the holding of the internal state of each handler. But each handler is purely functional and sequential.

Hmmm. Scalability in my experience comes from making the components of a system as self-contained as possible and constraining the interactions between them to the essentials.

How will this become easier if sometimes I'm using one abstraction mechanism (where it seems "paradigms" differ) and sometimes another?

Or perhaps the whole concept of "paradigm" in this context is starting to creak under the weight of semantic overload, and in fact what you are advocating is a new coherent approach that draws on elements and techniques from previous approaches?

To me this would be quite different from the mishmash of jostling features that usually goes under the name "multi-paradigm", and which essentially equates with "no paradigm".

A multiparadigm language can be well designed or poorly designed. A well designed one has a wide variety of programming concepts that are factored, which means that you can pick the ones you need and ignore the others. Depending on which ones you pick, you have different expressiveness and a different set of reasoning and programming techniques. Then a "paradigm" is just a particular set of concepts that happens to be useful for many problems.

See the Preface and Appendix D of CTM.

On the Expressive Power of Programming Languages

Thanks for the reference; I didn't know about this paper. If this example and the virtues of state are so well-known, then why is state still considered as "evil" or "awkward" or "impure" by functional programmers?

By that definition, all general purpose programming languages are "multi-paradigm" ;-)

I'm starting to think that "multi-paradigm" either means experimenting with mixing "your peanut butter with my chocolate", i.e. two things that are happend to be considered incompatible for purely historical reasons, or it means nothing at all.

Kinda leaning on the latter... ;-)

It's a question of degree. The concepts have to be factored: you can pick the ones you need without being forced to use the others. That's the key. There also have to be enough concepts that it makes a difference. It's technically correct to call C++ a multiparadigm language but in practice the multiparadigm aspect of C++ is very slim indeed!

I'm starting to think that "multi-paradigm" either means experimenting with mixing "your peanut butter with my chocolate", i.e. two things that are happend to be considered incompatible for purely historical reasons, or it means nothing at all.

Yes, if I understand your point, I think you're right: using concepts together that were considered incompatible for historical reasons.

In my view, the only reasonable language to use is a (well designed) multiparadigm language, since it lets you express what you want without getting in your way.

mixing "your peanut butter with my chocolate"
Hmmmm Postmodern?

Perhaps I'm missing something, but I don't see what the problem is here. If the state is an abstract data type, then the modules cannot depend on how the state is represented. So if we change the type that represents the state from State to (State, Counter), nothing about B, C, or D changes. If we add new operations getCounter and setCounter (of type m Counter and (Counter -> m ()), where m is the underlying monad), then A and E can use these to manipulate the counter, and the counter is simply ignored by B, C and D.