The Max-Abstraction Impulse, and Everything Else Wrong with Type-Level Genericity

These were my comments on RFC-1124 from F# 7.0, Interfaces With Static Abstract Methods, in the “Drawbacks Section”. It forms an essay on everything wrong with this particular form of Statically Constrained Genericity, and many of the things wrong with all the other forms.

Drawbacks

This feature sits uncomfortably in F#. Its addition to the .NET object model has been driven by C#, and its use in .NET libraries, and thus consuming and, to some extent, authoring IWSAMs is necessary in F#. However there are many drawbacks to its addition, documented below.

General drawbacks

Statically-constrained qualified genericity is strongly distortive of the practical experience of using a programming language, whether in personal, framework-building, team or community situations. The effect of these distortions are well known from:

• Standard ML in the 1990s (e.g. SML functors and “fully functorized programming”)
• C++ templates
• Haskell (type classes and their many technical extensions, abstract uses, generalizations and intensely intricate community discussions)
• Scala (implicits)
• Swift (traits)

While sometimes beautiful and simple as language design elements, these features are contested in programming communities. Their presence is deeply attractive to programmers who desire them – in many cases they become part of the fundamental organisational machinery applied to code and composition. They are equally problematic for those who don’t see overall value in the complexity their use brings. We should also note the relative success, simplicity and practical productivity of languages that omit these features including Elm, Go and even Python.

.NET has historically avoided this space. This was a deliberate decision in C# 2.0 by Anders Hejlsberg, who rejected proposals for statically-constrained genericity on the grounds of the complexity introduced v. the benefit achieved – a decision the initial author of this RFC (Don Syme) was involved with and agreed with. C# 11 and .NET 6/7 has since revisited this decision, primarily because reflective code of any kind is now considered more expensive in static compilation scenarios, and in C# reflection had frequently been used as a workaround for the absence of qualified genericity (other practical workarounds are available in F#, including the use of SRTP).

The following summarises the well-known drawbacks of these features, based on the author’s experience with all of the above. Emphatic language is used to act as a corrective.

Encouraging the Max-Abstraction impulse. Features in this space can encourage the practice of “max-abstraction” in C# and F# – that is, using more and more abstraction (in this case over types constrained by IWSAMs) to try to get maximal code reuse, even at the expense of code readability and simplicity. This kind of programming can be enormously enjoyable – it seems to satisfy a powerful desire in the human mind to abstract and generalise, and almost never loses its attraction. However, these techniques can also be an enormous waste of time, as very often the amount of code successfully reused is small, while the complexity in learning, comprehending, using, debugging and code-reviewing the corresponding frameworks is high, and the frameworks are often fragile.

Subsequent demands for more type-level computation. Features in this space can lead to demands for more features for type-level computation, giving ever more obscure code that is abstract, general and impenetrable. This in turn can give demand for more abstraction capabilities in the language. These will in turn feed the productivity-burning bonfire of max-abstraction.

Subsequent demand for compiler support for type-level debugging, profiling etc. Features in this space can lead to requests for tooling to support compile-time type-level computation (compile-time debugging, profiling etc.). They can also be very difficult to debug at runtime too, due to the numerous indirections and concepts encountered in even simple generic code. Being landed in generic code using TSelf: IMultiplicativeIdentity<TSelf,TResult> may not be intuitive to a beginner programmer trying to multiply a number by one.

A proliferation of micro-interfaces. For the specific case of IWSAMs, the use of nominal, declared interfaces means there will be a proliferation of interfaces, even to the granularity of one fully-generic interface for each method. The need to implement these explicitly will make the feature “rigid” and there will be complaints that the .NET Framework is not regular enough – that not enough interfaces are defined to allow maximally generic code, and that not enough types implement those interfaces. Knowing the role and meaning of these interfaces is a cost inflicted on more or less every .NET programmer – they can occur in error messages, and discussed in code review.

No stable point in library design. For the specific case of IWSAMs, how many micro-interfaces are “enough”? And how generic are they? In truth there can never be enough of these – ultimately you end up with one method for every single categorizable concept in the entire .NET Framework. The appetite for such abstractions is never-ending, and risks defeating other reasonable goals in software engineering.

The ‘correct’ degree of genericity becomes contested. Features in this space can bring a cultural attitude that “code is better if it is made more generic”. Some people will even claim such code is ‘simpler’ – meaning nothing more that it can be reused, and ignoring the fact that it is by no measure cognitively simpler. In code review, for example, one reviewer may say that the code submitted can be made more generic by constraining with the IBinaryFloatingPointIeee754<T> interface instead of INumber<T> – regardless of whether this genericity is needed. Another reviewer may have different point of view,and suggest a refactoring into two parts, one constrained by ILogarithmicFunctions<T> and IDecrementOperators<T>, another suggests switching to ISubtractionOperators<TSelf, TOther, TResult>. Note there is no effective measure of “goodness” here except max-abstraction. The end result is that the amount of genericity to use becomes unproductively contested.

As an aside, for F# SRTP code, the degree of genericity is automatically computed. For IWSAMs, in type-inferred languages with HM-type inference adjusting the amount of genericity is relatively simple. But the contested nature of generic code is still not productive outside of framework design.

Compiler and tooling slow-downs on large interface lists. For the specific case of IWSAMs, the presence of enormous generic interface lists can cause compiler slow-down. No one ever expected such lists of interfaces in .NET, this is a volcano of hidden complexity lying under the simple type double or decimal. [3/6/2024: as prophesized, an example of how this feature led to this in real life can be seen here]

To show that the above drawbacks are not imagined, consider the System.Double type in .NET 7. This now has the following list of interfaces:

public readonly struct Double :
    IComparable<double>
    IConvertible
    IEquatable<double>
    IParsable<double>
    ISpanParsable<double>
    System.Numerics.IAdditionOperators<double,double,double>
    System.Numerics.IAdditiveIdentity<double,double>
    System.Numerics.IBinaryFloatingPointIeee754<double>
    System.Numerics.IBinaryNumber<double>
    System.Numerics.IBitwiseOperators<double,double,double>
    System.Numerics.IComparisonOperators<double,double>
    System.Numerics.IDecrementOperators<double>
    System.Numerics.IDivisionOperators<double,double,double>
    System.Numerics.IEqualityOperators<double,double>
    System.Numerics.IExponentialFunctions<double>
    System.Numerics.IFloatingPoint<double>
    System.Numerics.IFloatingPointIeee754<double>
    System.Numerics.IHyperbolicFunctions<double>
    System.Numerics.IIncrementOperators<double>
    System.Numerics.ILogarithmicFunctions<double>
    System.Numerics.IMinMaxValue<double>
    System.Numerics.IModulusOperators<double,double,double>
    System.Numerics.IMultiplicativeIdentity<double,double>
    System.Numerics.IMultiplyOperators<double,double,double>
    System.Numerics.INumber<double>
    System.Numerics.INumberBase<double>
    System.Numerics.IPowerFunctions<double>
    System.Numerics.IRootFunctions<double>
    System.Numerics.ISignedNumber<double>
    System.Numerics.ISubtractionOperators<double,double,double>
    System.Numerics.ITrigonometricFunctions<double>
    System.Numerics.IUnaryNegationOperators<double,double>
    System.Numerics.IUnaryPlusOperators<double,double>

At this point, any reader should stop to consider carefully the pros and cons here. Each and every new interface adds conceptual overhead, and what was previously comparatively simple and compelling has become complex and curious. This complexity is potentially encountered by any and all users of .NET – beginner users trained in abstract math seem particularly fond of such numeric hierarchies, and are drawn to them like a moth to the flame. Yet these abstractions are useful only to the extent that writing generic code is successfully and regularly instantiated at many types – yet this is not known to be a significant real-world limiting problem for .NET today in practice, and for which many practical, encapsulated workarounds exist.

Drawback – Interfaces with static abstract methods will get misunderstood as types, not type-constraints

Consider this code:

let addThem (x: INumber<'T>) (y: INumber<'T>) = x + y

This code will not compile, and example error for corresponding C# code is shown below.

To understand why, note that the relevant static abstract method in the INumber<'T> hierarchy is this (simplified):

type IAdditionOperators<'T when 'T : IAdditionOperators<'T>> =
    static abstract (+): x: 'T * y: 'T -> 'T

The operator takes arguments of type 'T, but the arguments to addThem are of type INumber<'T>, and not 'T ('T implies INumber<'T> but not the other way around). Instead the code should be as follows:

let addThem (x: 'T) (y: 'T) when 'T :> INumber<'T> = x + y

This is really very, very subtle – beginner users are often drawn to generic arithmetic, and any beginner will surely think that INumber<'T> can be used as a type for a generic number. But it can’t – it can only be used as a type-constraint in generic code. Perhaps analyzers will check this, or special warnings added.

Fortunately in F# people can just use the simpler SRTP code on most beginner learning paths:

let inline addThem x y = x + y

Drawback – Type-generic code is less general than explicit function-passing code

Type-generic code relying on IWSAMs (and SRTP) can only be used with types that satisfy the constraints. If the types don’t satisfy, you have a lot of trouble.

type ISomeFunctionality<'T when 'T :> ISomeFunctionality<'T>>() =
    static abstract DoSomething: 'T -> 'T

let SomeGenericThing<'T :> ISomeFunctionality<'T>> arg = 
    //...
    'T.DoSomething(arg)
    //...

type MyType1 =
    interface ISomeFunctionality<MyType1> with
        static member DoSomething(x) = ...

type MyType2 =
    static member DoSomethingElse(x) = ...

SomeGenericThing<MyType1> arg1
SomeGenericThing<MyType2> arg2 // oh no, MyType2 doesn't have the interface! Stuck!

When the number of methods being abstracted over is small (e.g. up to, say, 10) an alternative is to do away with the IWSAMs and simply to pass a DoSomething function explicitly:

let SomeGenericThing doSomething arg =
    //...
    doSomething arg
    //...

with callsites:

type MyType1 =
    static member DoSomething(x) = ...

type MyType2 =
    static member DoSomethingElse(x) = ...

SomeGenericThing MyType1.DoSomething arg1
SomeGenericThing MyType2.DoSomethingElse arg2

Note that explicit function-passing code is shorter and more general – it works with both MyType1 and MyType2. In F# this kind of code is incredibly safe and succinct because of Hindley-Milner type inference – passing functions and making code generic are two of the very easiest things to do in F#, the language is almost made for exactly those activities.

For the vast majority of generic coding in F# explicit function-passing is perfectly acceptable, with the massive benefit that the programmer doesn’t burn their time trying to create or use a cathedral of perfect abstractions. SRTP handles most other cases.

Drawback – Implementations of static abstract methods are not parameterizable, they can’t close over anything

F# is driven by explicit parameterization, for example functions:

let someFunction x = 
   ...x...

or classes:

type SomeClass(x) = 
   ...x...

These constructs are explicitly parameterizable and capture their parameters: whatever they return – or the methods they provide – can close over arbitrary new dependencies by adding them to the parameter lists. For example:

let someFunction newArg x = 
   ...x...newArg...

or classes:

type SomeClass(newArg , x) = 
   ...x...newArg...

This is at the heart of F# programming and all functional programming, and it is powerful and accurate. Additionally, requirements change: what is initially independent may later need to become dependent on something new. In F#, when this happens, 99% of the time you plumb a parameter through: perhaps organising them via tuples, or records, or objects. Either way the adjustments are relatively straight-forward. That’s the whole point.

It is obvious-yet-crucial that implementations of static abstract methods have a fixed signature and are static. If an implementation of a static abstract method later needs something new – something unavailable from the inputs or global state or implicit thread/task context – you are stuck. Normal static methods can become instance methods in this situation, or take additional parameters. But implementations of static abstract methods can’t become instance, and they can’t take additional explicit parameters: since they must be forever static and must always take specific arguments. You are stuck: you literally have no way of explicitly plumbing information from A to B. Your options are switching to a new set of IWSAMs (plus a new framework to compose them), or removing the use of IWSAMs and returning to the land of objects and functions.

To see why this matters, consider a basic IParseable<T> (the actual IParseable<T> has some additional options, see below).

type IParseable<'T> =
    static abstract Parse: string -> 'T

So far so good. Now let’s assume you have a set of 100 domain classes implementing IParseable<T> and you’ve written a framework to compose these, e.g. extension methods to generically parse arrays and grids of things, read files and so on – whatever the guidance says you’ve gone deep, really deep into the whole IParseable<T> thing and it feels good, really good. Each implementation looks like this:

type C =
    interface IParseable<C> with
        static abstract Parse(input) = ....

Now assume it’s the last night of your project – you’re submitting your code tomorrow – and the specification of your parsing changes so that, for several types your parsers now need to selectively and dynamically parse multiple versions at different points in the data stream – say v1 and v2. You try to write this:

type C =
    interface IParseable<C> with
        static abstract Parse(input) =
            if version1 then 
                ....
            else
                ....

But how to get version1 into Parse? In this case, there is literally no way to explicitly communicate that parameter to those two implementations of IParseable<T>. This means your composition framework built on the IWSAM IParseable<T> may become useless to you, due to nothing but this one small (yet predictable) change in requirements. You will now have to remove all use of IParseable<T> and shift to another technique, or else rely on implicit communication of parameters (global state, thread locals…). This is no theoretical exercise – format parsers often change, by necessity over time, and need variations.

What’s gone wrong? Well, IWSAMs should never have been used for parsing domain objects – and a compositional framework should never use IWSAMs. But IParseable<'T> sounded so compelling to implement, didn’t it? And those IWSAMs combining them were so mathematically elegant and beautiful, weren’t they? Well, none of that matters now. Specifically, IWSAMs like IParseable<T> should only be implemented on types where the parsing implementation is forever “closed” and “incontrovertible”. By this we mean that their implementations will never depend on external information (beyond culture/date/number formatting, see below), and there will be no variations on how the implementations should act.

What should you do instead? Well, you should always have used regular interfaces, objects and functions – the world of normal functional-object programming – that is, parser combinators. For composition you should use explicit composition of parser functions and objects. All this is standard functional-object programming. For example:

type Parser<'T> = Parser of (string -> 'T)

module Parsers =
    let SomeClassParser1 = Parser (fun s -> ...)

    let SomeClassParser2 = Parser (fun s -> ...)

    let SomeClassParser version =
        if version = V1 then SomeClassParser1 else SomeClassParser2

With this approach you can write and compose parsers happily – the parsers are first-class objects. You can also lift any value of type T :> IParseable<T> into a Parser and then compose happily. For example:

module Parsers =
    let LiftParseable<'T when IParseable<'T>> =
        Parser p.Parse

    let IntParser = LiftParseable<int>
    let DoubleParser = LiftParseable<double>

    let OverallParser = CombineParsers (IntParser, DoubleParser, SomeClassParser1)

See FParsec for a full high-performance compositional parser framework.

This doesn’t make implementing IParseable<'T> wrong – it’s just very important to understand that you should only implement it on types where the parsing implementation is forever “closed” and “incontrovertible”, as defined above. And it’s very important to understand that your composition frameworks should not use IWSAMs as the primary unit of composition. See “Guidance” below.

Another way of looking at this is that, unlike actual functions and objects, IWSAM implementations live and are composed at a different “level” than the rest of application – the level of static composition with generics – they are immune to regular parameterization. This means using IWSAMs has some of the same characteristics as original Standard ML functors or C++ templates, both of which partition software into two layers – the core language and the composition language. Another way to look at it is to note that IWSAMs are not a first class thing – a function or method can’t return an IWSAM implementation.

Note: IParseable does have an implicit way of passing extra information, through the optional IFormatProvider argument. However in practice that’s pretty much nightmare-unusable for such purposes as propagating information – using an bespoke IFormatProvider is notoriously difficult and these are best left only for numeric, date and culture formatting parameters. So in this section we are discussing explicit parameterization of additional information that informs the action of parsing domain objects.

Note: It could be said “IParseable is only for parsing numbers, dates and so on, and it’s simply not for parsing domain objects where parsing different variations requires significant code”. That’s fine. But you can see why people might think otherwise.

Note: As pointed out on twitter it is possible to plumb “statically known” information to an IWSAM implementation by passing additional IWSAM constrained type parameters. This means doubling down on more type-level parameterization, and any further IWSAMs passed are subject to the same problems.

Drawback – Three ways to abstract

In F# there are now three mechanisms to do type-level abstraction: Explicit function passing, IWSAMs and SRTP.

Within F#, explicit function passing should generally be preferred. SRTP and IWSAMs can be used as needed.

Explicit function passing:

let f0 add x y =
    (add x y, add y x)

SRTP

let inline f0<^T when ^T : (static member Add: ^T * ^T -> ^T)>(x: ^T, y: ^T) = 
    let res1 = 'T.Add(x, y)
    let res2 = 'T.Add(x, y)
    res1, res2

NOTE: this simplified syntax is part of this RFC

IWSAMs

type IAddition<'T when 'T :> IAddition<'T>> =
    static abstract Add: 'T * 'T -> 'T

let f0<'T when 'T :> IAddition<'T>>(x: 'T, y: 'T) =
    let res1 = 'T.Add(x, y)
    let res2 = 'T.Add(y, x)
    res1, res2

These have pros and cons and can actually be used perfectly well together:

Guidance

A summary of guidance from the above:

• Understand the inherent limitations of IWSAMs. IWSAM implementations are not within the “core” portion of the F#: they are not first-class objects, can’t be produced by methods and, can’t be additionally parameterized. IWSAM implementations must be intrinsic to the type, they can’t be added after-the-fact.
• Don’t give into type-categorization impulse. With IWSAMs, you can happily waste years of your life carefully categorising all the concepts in your codebase. Don’t do it. Throw away the urge to categorise. Forget that you can do it. It almost certainly isn’t helpful to categorise the types in your application code using these.
• Don’t give into max-abstraction impulse. With IWSAMs and other generic code, you can happily waste even more years of your life max-abstracting out every common bit of code across your codebase. Don’t do it. Throw away the urge to max-abstract just for its own sake. Forget that you can do it, and if you try don’t use IWSAMs, since using explicit function passing will likely result in more reusable generic code (see below).
• Using IWSAMs in application code carries a strong risk you or your team will later remove their use. Explicitly plumbing new parameters to IWSAM implementations is not possible without changing IWSAM definitions. Because of this, using IWSAMs exposes you to the open-ended possibility that you will have to use implicit information plumbing, or remove the use of IWSAMs. Given that F# teams generally prefer explicit information plumbing, teams will often remove the use of devices that require implicit information plumbing.
• Only implement IWSAMs on types where their implementations are stable, closed-form, and incontrovertible. IWSAMs work best on highly stable types and operations where there is essentially no future possibility of requirements changing to include new parameters, dependencies or variations of implementation. This means that you should only implement IWSAMs on types where their implementation is forever “closed” and “incontrovertible” – that is, unarguable. Numerics are a good example: these types and operations are highly semantically stable, require little additional information and have very stable contracts. However your application code almost certainly isn’t like this.
• Do not use IWSAMs as the basis for a composition framework. You should not write composition frameworks using IWSAMs as the unit of composition. Instead, use regular programming with functions and objects for composition, and write helpers to lift leaf types that have IWSAM definitions into your functional-object composition framework. See examples above.
• Prefer explicit function passing for generic code. In F# there are now three mechanisms to do type-level abstraction: Explicit function passing, IWSAMs and SRTP. Within F#, when writing generic code, explicit function passing should generally be preferred. SRTP and IWSAMs can be used as needed. See examples above.
• Go light on the SRTP. Some of the changes in this RFC enable nicer SRTP programming. Some of the above guidance applies to SRTP – for example do not use SRTP as a composition framework.
• For generic math, use SRTP or IWSAM. Generic math works well with either. F# SRTP code is often quicker to write, requires less thought to make generic and has better performance due to inlining. If C#-facing, use IWSAMs for generic math code.
• For generic math using units-of-measure, use SRTP. The .NET support for generic math does not propagate units of measure correctly. Rely on F# SRTP code for these.
• If defining IWSAMs, put static members in their own interface. Do not mix static and non-static interfaces in IWSAMs.