22
$\begingroup$

What can programming languages do to make unit testing easier?

For the purposes of this question, I'm especially interested in unit testing procedural code with lots of heavy side effects.

Let me define a heavy side effect as one like writing to disk or calling an external service and specifically not mutating a data structure. Basically, a side effect that one would not want in a unit test but would want in prod is heavy.

I spend a lot of time working with Go and Python, two languages that heavily use unit tests as a cultural practice.

I've noticed that writing code that's easy to read and code that's easy to unit-test seem to be somewhat conflicting goals and I'm wondering if there's a way around this by designing languages differently.

In particular, the problem I run into is that procedural code is extremely easy to read and understand but difficult to unit test because you need to transfer control to the non-prod implementation at some point.

In Go, I think people usually solve this problem with interfaces. I use a lot of SomeServiceClient interfaces, where SomeServiceClient has one implementation that's used in prod and maybe a handful of in-memory implementations that I use for unit tests.

In Python, I use mock.patch liberally to stub out function calls with heavy side effects. This is good for keeping the code readable, but it requires a lot of knowledge of the internals of a function, and it prevents tests from being run in parallel. It's also possible to get it wrong and forget to stub out a function with a heavy side effect. mock.patch also directly modifies the global Python module symbol table thing, so it is very sensitive to how a symbol was imported in the module under test. I suspect that mock.patch as a language feature rather than a library feature could be designed to avoid these drawbacks. For example, functions could be organized into groups, and stubbing out one of them in a particular scope could cause the others in the same group to be stubbed out as well.

I'm also dimly aware that in languages like C, you can use the build system to substitute test versions of dependencies at build time, but I don't have firsthand experience with maintaining a project that uses this strategy.

$\endgroup$
3
  • 1
    $\begingroup$ In Minecraft, we have MockBukkit which replaces all the impls of Bukkit/Spigot interfaces with stubs $\endgroup$
    – Seggan
    Aug 16, 2023 at 14:37
  • $\begingroup$ Does python's included libraries count as a language feature? I find pytest to be easier to use than builtin unittest. $\endgroup$
    – qwr
    Aug 16, 2023 at 21:15
  • 1
    $\begingroup$ I have advocated among my friends - with no acceptance :-(! - a breaking encapsulation keyword for C++ - similar to friend but in reverse: you put it in the class you want to have access to the protected_/_private fields of another class. Which is often what you want in a unit test: access to a classes' internals without going to contortions and without exposing those internals to anyone. Then you could confirm that it was only used appropriately by grepping for that string in your codebase - if it appears outside of a unit test directory then flag it as a style violation. $\endgroup$
    – davidbak
    Aug 17, 2023 at 22:53

6 Answers 6

15
$\begingroup$

Built-in support

Built-in support for tests makes it easier to add tests.

It seems basic, but in C or C++ adding tests means adding an extra binary, likely with a test framework or utility dependencies, and that's a significant enough hurdle that many projects have no test at all.

Lowering the barrier to entry, by making it as easy as possible to add tests, makes testing much easier.

Accessibility Backdoor

Unit-tests may require testing pieces of code that are NOT exposed in the API of the library.

In C or C++, only what is exposed in the API of the library can be tested, which regularly leads to over-exposing, conditionally exposing when building with a special build flag, or skipping testing.

Built-in support for tests to have access to non-exposed pieces of code makes it easier to test small units.

Polymorphism

Unit-tests typically focus their tests on a specific piece of code. If one first needs to build an entire world for this unit of code to operate, creating and maintaining the test will be very, very, heavy.

By allowing the user to have code depend on interfaces rather than concrete types, the language allows substituting spies, mocks, dummies, etc... and therefore avoiding having to build the world.

Note that polymorphism can be provided in may ways, depending on the language: dynamic types, interfaces, templates, generics, ... all offer forms of polymorphism.

I can already hear arguments that overuse of polymorphism may lead to relying on implementation details of the unit under test. That is true, but that is arguably not the fault of polymorphism: bad tests in any case.

Compile-Time Tests

Unit-tests in a statically compiled language should provide a way to test that code does not compile.

In a statically compiled language, types are regularly used to enforce invariants at compile-time... but who watches the watchers? In order to make sure that the types are properly setup to prevent certain uses, a test is necessary!

Ensuring tests does not compile, and does not compile for the right reason, has little support in mainstream languages, unfortunately. It's not an easy problem, admittedly, but it certainly is an important one: no run-time test can substitute, after all!

$\endgroup$
6
$\begingroup$

Not specifically targeted for easier unit testing, the following are techniques in various languages to support swappable effect implementations.

The common pattern in all of these is to make effects explicit and part of the function signature: instead of

  • readFile(Path) -> String
  • writeFile(Path) -> Unit

The functions have signatures like

  • readFile(Path) -> Read<String>
  • writeFile(Path) -> Write<Unit>

or

  • readFile(Path) -> String with Read
  • writeFile(Path) -> Unit with Write

Algebraic Effects and handlers

Algebraic effects are fairly new and complicated, so as of now most languages don't support them. Languages built with algebraic effects in mind are Koka, Unison, and Eff; and OCaml recently got limited support.

Example in Eff (note: doesn't compile but shows the idea):

type path = string

(* Define effects *)
effect ReadFile: path -> string
effect WriteFile: path -> string -> unit

(* Define implementations *)
let prod_read_write = handler
| effect (ReadFile path) k -> continue k (perform (Read path))
| effect (WriteFile path contents) -> continue k (perform (Write path contents))
;;
let mock_read_write initial_state = handler
| y -> (fun s -> Right (y, s))
| effect (ReadFile path) k -> (fun s -> match assoc_opt path s with
    | None -> Left (path ^ " not found")
    | Some x -> continue k x s)
| effect (WriteFile path contents) k -> (fun s -> continue k () (update path contents s))
| finally f -> f initial_state
;;

(* Define operation *)
let complex_operation =
    let foo_contents = perform (ReadFile "foo.txt") in
    let bar_contents = perform (ReadFile "bar.txt") in
    perform (WriteFile "baz.txt" (foo_contents ^ bar_contents))
    ;;

(* Test operation *)
let test_complex_operation =
    let initial_state = [("foo.txt", "foo"), ("bar.txt", "bar")] in
    let final_state = with mock_read_write initial_state handle complex_operation in
    match final_state with
    | Left message -> error ("unexpected error: " ^ message)
    | Right final_state -> assert (assoc_opt "baz.txt" final_state = Some "foobar")
    ;;

test_complex_operation

Tagless Final

Tagless final is another way to represent effects in a way that makes them mockable.

Tagless Final is common in Scala, but also possible in Haskell. In fact, mtl (monad transformers as typeclasses) is essentially Tagless Final plus a limited form of composition (e.g. MonadReader is the tagless final Reader, while ReaderT m implements MonadReader as well as any Monad... classes. But ReaderT m only implements Monad... classes defined in mtl and you must give it your custom Monad... class implementations manually; this a big problem in mtl, as well as performance issues).

Example in Scala:

import java.nio.file.{Files, Path, Paths}
import scala.collection.mutable.Map
import cats.Monad
import cats.data.StateT
import cats.implicits._

// Define effects
trait Reader[F[_]] {
  def read(path: Path): F[String]
}
trait Writer[F[_]] {
  def write(path: Path, content: String): F[Unit]
}

// Define the production implementation
type Id[A] = A
implicit object FileSystemReader extends Reader[Id] {
  def read(path: Path): String = Files.readString(path)
}
implicit object FileSystemWriter extends Writer[Id] {
  def write(path: Path, content: String): Unit = Files.writeString(path, content)
}

// Define the mock implementation, with hidden state
type ErrorOr[A] = Either[String, A]
type MockState[A] = StateT[ErrorOr, Map[Path, String], A]
implicit object MockReader extends Reader[MockState] {
  def read(path: Path): MockState[String] = StateT { state =>
    state.get(path) match {
      case None => Left(path.toString() ++ " not found")
      case Some(x) => Right((state, x))
    }
  }
}
implicit object MockWriter extends Writer[MockState] {
  def write(path: Path, content: String): MockState[Unit] = StateT { state =>
    state.put(path, content)
    Right((state, ()))
  }
}

// Define the complex operation
def complexOperation[F[_] : Monad](
  implicit reader: Reader[F], writer: Writer[F]
): F[Unit] = for {
  fooContents <- reader.read(Paths.get("foo.txt"))
  barContents <- reader.read(Paths.get("bar.txt"))
  _ <- writer.write(Paths.get("baz.txt"), fooContents ++ barContents)
} yield ()

// Test using the mock state
def testComplexOperation(): Unit = {
  val initialState = Map(Paths.get("foo.txt") -> "foo", Paths.get("bar.txt") -> "bar")
  complexOperation[MockState].run(initialState) match {
    case Left(message) => throw Error("unexpected error: " ++ message)
    case Right((finalState, ())) => assert(finalState.get(Paths.get("baz.txt")) == Some("foobar"))
  }
}

testComplexOperation()

Freer Monads (2)

A freer monad is another way to represent effects. Essentially, map, pure, and join are converted into data-structures, so your function builds a giant AST-like structure which can then be run by different interpreters depending on whether you are testing or running in production.

It's similar to tagless final, but all-but-deprecated by the former because it's very slow: freer monad operations build up and then consume a large data structure with many delayed computations, while tagless final operations run instantly.

Example in Haskell:

{-# LANGUAGE Edition2021 #-}
import Prelude
import Data.Map
import Control.Monad.State
type Path = String

-- The freer monad definition, this isn't program-specific and could be in a library
data Freer f a where
    Pure :: a → Freer f a
    Impure :: f b → (b → Freer f a) → Freer f a

instance Functor (Freer f) where
    fmap f (Pure x) = Pure (f x)
    fmap k' (Impure fy k) = Impure fy (fmap k' . k)

instance Applicative (Freer f) where
    pure = Pure
    FPure f      <*> x = fmap f x
    FImpure fy k <*> x = FImpure fy ((<*> x) . k)

instance Monad (Freer f) where
    return = Pure
    Pure x      >>= f = f x
    Impure fy k >>= k' = Impure fy (k' <<< k)

lift :: f a -> FFree f a
lift fx = FImpure fx FPure

-- How to use
data ReadWrite a where
    Read :: Path -> ReadWrite String
    Write :: Path -> String -> ReadWrite ()

runReadWriteProd :: FFree ReadWrite a -> IO a
runReadWriteProd (FPure x) = pure x
runReadWriteProd (FImpure fy k) = do
    x <- case fy of
        Read path -> readFile path
        Write path contents -> writeFile path contents
    k x
runReadWriteMock :: FFree ReadWrite a -> StateT (Either String) (Map Path String) a
runReadWriteMock (FPure x) = pure x
runReadWriteMock (FImpure fy k) = do
    x <- case fy of
        Read path -> \s -> case lookup path s of
            Nothing -> Left $ path ++ " doesn't exist"
            Just x -> Right (s, x)
        Write path contents -> \s -> Right (insert path contents s, ())
    k x
type ProdReadWrite = IO
type MockReadWrite = State (Map Path String)

complexOperation :: FFree ReadWrite ()
complexOperation = do
    foo <- lift $ Read "foo.txt"
bar <- lift $ Read "bar.txt"
    lift $ Write "baz.txt" $ foo ++ bar

testComplexOperation :: ()
testComplexOperation =
    let initialState = fromList [("foo.txt", "foo"), ("bar.txt", "bar")] in
    let finalState = runStateT initialState $ runReadWriteMock $ complexOperation initialState in
    case finalState of
        Left message -> error $ "unexpected error: " ++ message
    Right (finalState, ()) -> assert $ lookup "baz.txt" finalState == "foobar"

main :: IO ()
main = testComplexOperation `deepseq` pure ()

Module Functors

Module functors are essentially swappable imports for your modules. So you can make the module which you import the effects from into a parameter, and then when testing, instantiate the module with a mock effect handler.

This is less flexible for the above (see: no managing internal state and exception handling), but easier for the implementer, since there are no monad comprehensions (do in Haskell and for in Scala).

It's uncommon and a bit ugly to do this in OCaml, but perhaps a language will implement it with better syntax.

type path = string
module MyCode (Read : sig
    val read : path -> string
end, Write : sig
    val write : path -> string -> unit
end) = struct
    fun complex_operation () =
        let foo = read "foo.txt" in
        let bar = read "bar.txt" in
        write "baz.txt" (foo ^ bar)
        ;;
end

(* Only test implementation shown *)
open MyCode(struct
    fun read = function
    | "foo.txt" -> "foo"
    | "bar.txt" -> "bar"
    | path -> error (path ^ " not found")
    ;;

    fun write path contents = match (path, contents) with
    | ("baz.txt", "foobar") -> ()
    | _ -> error ("unexpected path \"" ^ path ^ "\" or contents \"" ^ contents ^ "\"")
    ;;
end)

complex_operation ()
$\endgroup$
2
  • $\begingroup$ This reply is interesting: The question asks about language support for testing, but every of suggestions in this reply requires the programmer to rewrite code, so it is not about language support. Arguably, all program rewrites suggested here make the program less readable for mainstream imperative programmers, so just what the original question wants to avoid. $\endgroup$ Aug 17, 2023 at 14:32
  • $\begingroup$ @MartinBerger: Language support always requires source-level changes. If you were trying to avoid those, you'd be talking about improved tooling for existing languages, not design of improved languages. $\endgroup$
    – Ben Voigt
    Aug 17, 2023 at 21:51
5
$\begingroup$

Use callbacks for side-effects

A function which writes to a file or network socket can be transformed into one which writes to a "writer" function or object, where the "writer" is an additional argument. Similarly, a function which makes HTTP requests can be transformed into one which makes requests to a callback function or object. These functions can then be tested by passing dummy callbacks which just check that the correct data is sent or the correct request is made.

This means that the parts of the code which supply the "real" callbacks (the ones that actually perform the side-effects) can't be unit tested, but those parts of the code can be kept small, simple and "obviously correct" enough that they don't need to be. In the extreme case, a whole application might pass around these callbacks everywhere, and only need to be provided with the "real" ones at one entry-point.

So in terms of language design, if you make it easy to pass these callbacks around ─ particularly through multiple levels of function calls ─ without having to write a lot of boilerplate code to manually pass them further along, then such applications will be easier to test.

Represent side-effects as data

Many programs read all of their input before writing any output; any such program can be rewritten as a pure function which accepts a string or byte array, and returns a string or byte array. Being a pure function, it is straightforward to unit test; even in imperative languages, it is possible (and beneficial) to write pure functions, and they can be "internally impure" as long as they aren't observably impure from the outside.

In other applications, output side-effects will be interleaved with input side-effects, so that a program must produce some output before it can accept further input. But it's generally possible to move the side-effects further up the call graph, closer to the entry-point, making more of the codebase pure (at least, pure to outside observers). This is the "functional core, imperative shell" approach (see also this talk).

The main thing that can go wrong here is if a supposedly "pure" function leaks its internal impurity into the rest of the application. If this happens, the supposedly-pure function might pass all of its unit tests but still fail on the same inputs when called in a different context. So in the language design, then, it would help immensely if the compiler were able to verify that "internally impure" functions are actually pure. See this other Q&A for more discussion of this.

Besides that, it is a good idea to support immutable data structures for the data at the boundaries between internally-impure parts of the program. Make it easy to define data types with copy-on-write behaviour, for example.

$\endgroup$
6
  • $\begingroup$ This reply is interesting: The question asks about language support for testing, but every of suggestions in this reply requires the programmer to rewrite code, so it is not about language support. Arguably, all program rewrites suggested here make the program less readable for mainstream imperative programmers, so just what the original question wants to avoid. $\endgroup$ Aug 17, 2023 at 14:30
  • 3
    $\begingroup$ @MartinBerger I think you have not understood me. I start from the assumption that the programmer wants to write code that is easy to unit test, otherwise there is no sense in giving them features to make that easier to do. There are well-known design principles for making imperative code easy to unit test, so starting from those principles, it follows that a language designed to make code easier to test should encourage following those principles and provide features which support doing so. I am not sure how you missed the parts where I clearly referred to role of language design in this. $\endgroup$
    – kaya3
    Aug 17, 2023 at 14:38
  • 4
    $\begingroup$ The ways that people choose to write code are highly influenced by what the language makes easy or difficult, so it is unlikely that people writing in a language designed for making unit testing easy would still write in the same hard-to-test imperative styles. Consider error-handling; e.g. in Rust most error-handling is done by returning discriminated unions like Result<T, E> and Option<T>, as opposed to using sentinel values or by throwing and catching exceptions. Those other styles are possible in Rust, but people choose not to use them because the language supports a better way. $\endgroup$
    – kaya3
    Aug 17, 2023 at 14:45
  • 1
    $\begingroup$ I don't agree. Those "well-known design principles for making imperative code easy to unit test ..." are valuable for current imperative languages, but current imperative languages have no good built-in support for testing. (Haskell and Scala have implicit arguments which goes a bit in this direction). However, I don't think this is the end of the story. I interpret the question as: what could be PL support that future languages could have $\endgroup$ Aug 17, 2023 at 14:45
  • 1
    $\begingroup$ @MartinBerger We have both interpreted the question that way, and my answer directly addresses PL support in both sections. Your objection seems to be that I have determined what that support should be by thinking about the known ways of making unit testing easier. $\endgroup$
    – kaya3
    Aug 17, 2023 at 14:48
2
$\begingroup$

The language should allow to access, override and call also private methods and data structures, disregarding final on methods and classes, while in the test. This must be different from the main code where visibility rules are enforced.

The visibility architecture should be designed for the main code that runs in production. If the method is only called from inside the object, it is likely to be marked private. This does not mean that the method is very trivial, or its functionality changes a lot very often. A sum(a, b) must return a sum, not a sum plus one in one version, minus one in another. If it is not clear what the function is actually doing, and this keeps changing all the time, the functionality is likely divided wrongly and such a code does not look well maintainable.

When developing a complex algorithm, a developer may split it into simpler logical parts, each doing clear and documented step. It is quite logical to write a unit test for each step, and actually often done during development, just to see if the step is implemented well. But in the final version, these methods are only called from inside, so may become private, making them not accessible for the further testing. Some bigger test can be written that calls a public method, but if the output is different from expected, the reasons are then very difficult to trace. Also, in some cases it is really difficult to figure input that covers all test branches in the steps further downstream. Like, if one step computes a hash code of the string, and another finds the bucket number from this hash, it may be quite a work to forge strings with hash codes corresponding the first, last bucket, or dividable by 16, or something else that may get wrong. A direct test that calls the method to compute the bucket number from the hash code would be much easier to write, same as another that tests the known corner cases with given bucket numbers.

There are also objects that do something a proper test should not do, like interacting with remote database across the globe in they only public method. It may be reasonable to define a method that takes the query and reads everything from the database. A test could substitute/override such method to make the implementation testable, but it is otherwise not called from outside so would not need to be public (or all class may be declared final for some unrelated reason). Or you may design a "data provider" to talk with the database, but in order to inject a test mock instead of this provider, you still need either a public setter or a public constructor taking the provider. If production code only permits that it uses, there may not be enough public access.

I understand there is another option that "you should test the final output from the expected input" that is also common to hear. I hope I have explained the two reasons why there may be a different opinion of this. I have many years of programming experience, and all successful software I ever developed had extensive testing suites.

You can also look into some good mocking framework like Mockito, or some testing library beyond trivial testing functionality like gtest (death tests, etc). These testing libraries sometimes do incredible things beyond one could imagine impossible. Putting them right into the language would likely be easier and without unexpected limitations.

$\endgroup$
2
$\begingroup$

I'm also dimly aware that in languages like C, you can use the build system to substitute test versions of dependencies at build time, but I don't have firsthand experience with maintaining a project that uses this strategy.

Rust's conditional compilation makes this fairly easy, enough so that it's a common pattern. I think a good example is in the announcement for Turmoil, a testing library which provides drop-in replacement for tokio for mocking.

// This is only imported in test builds
#[cfg(test)]
pub use turmoil::net::*;

// This is imported in all builds except test builds
#[cfg(not(test))]
pub use tokio::net::*;

fn run_server() -> ! {
    // This refers to turmoil::net::tcp::TcpListener in test builds,
    // and tokio::net::tcp::TcpListener in all other builds
    let socket = tcp::TcpListener::bind("1.2.3.4:5678");
    ...
}

fn run_client_request(input: &str) -> Result<String, ClientError> {
    // Same as above, turnmoil::net::tcp::TcpSteram when testing and
    // tokio::net::tcp::TcpStream when running
    let socket = tcp::TcpStream::connect("1.2.3.4:5678");
    ...
}

// This code will only be built when testing
#[cfg(test)]
mod simulation {
    #[test]
    fn simulate_it() -> turmoil::Result {
        let mut sim = turmoil::Builder::new().build();
        sim.host("server", || async move {
            run_server();
        });

        sim.client("test", async move {
            let output = run_client("foo")?;
            assert_eq!(output, "bar");

            Ok(())
        });

        sim.run()
    }
}

In your Cargo.toml you will also need to configure the dependencies correctly:

...

[dependencies]
tokio = "1.32.0"
...

[dev-dependencies]
turmoil = "0.5.6"
...

If you run cargo test, cargo will build the project using turmoil and run the test. But if you run cargo build or cargo run, it will build using tokio. The common code builds on both because turmoil::net and tokio::net have the exact same API. This even works if the code is in a library and used as a dependency, as long as the dependent library follows the same pattern of importing turmoil in test builds and tokio otherwise, since the test flag propogates.

Alternatively, you could use #[cfg(feature = "turmoil")] and #[cfg(not(feature = "turmoil"))] instead of #[cfg(test)]and#[cfg(not(test))], as the Turmoil announcement does. In that case your Cargo.toml` may look something like:

...

[dependencies]
tokio = "1.32.0"
turmoil = { version = "0.5.6", optional = true }
...
$\endgroup$
1
$\begingroup$

Tyr has unit tests built into the language. Today, this allows two things that cannot be implemented by test libraries:

Tests have special visibility rules allowing them to access arbitrary entities within the same compilation unit, i.e. library. This allows tests to check internal state where the test should be able to do so while allowing the overall API to prevent access to that internal state.

Tests can get a noCompile attribute telling the compiler that the test is successful if compilation fails with a specific error message (example). The motivation is to write a test ensuring that internal state is not accidentally revealed by some change.

Finally, code only reachable through tests is removed by the compiler. This is, however, achievable with any language capable of performing a global dead code elimination.

$\endgroup$
5
  • $\begingroup$ This is interesting, but the test annotations are interspersed with the code itself. That intersperses test and code. Would it not be desirable to keep them radically separate, but still with language support? $\endgroup$ Aug 17, 2023 at 14:34
  • $\begingroup$ You can move tests to a different module and in fact are encouraged to if you want to write a test that is forced to used the public API. I honestly do not see why it would be a good idea to move tests away from the implementation. Like comments, they document the intended behaviour of your code and should not be separated and forgotten. $\endgroup$
    – feldentm
    Aug 17, 2023 at 17:28
  • $\begingroup$ How do external tests refer to the point they test from the outside? $\endgroup$ Aug 17, 2023 at 20:25
  • $\begingroup$ I'm not sure a language should force programmers to put in a certain place, I think code and tests are conceptually separate. For example random tests, or integration. $\endgroup$ Aug 17, 2023 at 20:26
  • $\begingroup$ That's not a contradiction. You can still have a test package and even the standard library does that for API tests. $\endgroup$
    – feldentm
    Aug 18, 2023 at 14:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .