Home > java > Is there such a thing as the NoMock movement?

Is there such a thing as the NoMock movement?

One of the things that’s often advocated for a good software development process is having unit tests.

The more of those tests we have, the better. Having them increases our confidence that our code works as we think it works, isolates issues quickly to the area where they are introduced and thus increases our ability to do refactoring.

Central to unit testing in object oriented systems is the concept of a “Mock”. Those software systems largely exist out of objects that collaborate with each other in some way. Objects having internal references to other objects are said to be having dependencies on those other objects.

In unit testing we would like to test the smallest possible unit, which represents a single code path through a single method. As soon as that code calls a dependent object, we can’t call it a unit test anymore. In order to restore balance, and make the world perfect again, all those dependencies should be replaced by fake objects, called Mocks. See e.g. When Should I mock.

For years we have learned that we all should do this. Unit tests are good, unit tests need mocks. Therefor we unit test, and therefor we mock. Life was simple.

Lately however a growing group of people seem out to disturb our perfect little world. Their creed: “Mocks are wrong. We should use real objects”. They don’t seem to have a name yet, but following the NoSQL movement, let’s call them the NoMock movement.

Among them are people like Bill Burke, Dan Allen, Andrew Lee Rubinger, Augie Fackler, Nathaniel Manista and Stan Silvert.

They spread their ideas in blogs and presentations, e.g.:

So what exactly did we thought mocks bought us and what parts of that are those NoMock guys rebutting?

One benefit of mocks that comes up often is performance. Surely mocks are much faster than using say a real EJB as a dependency that requires starting up a WebSphere server that takes 30 minutes for a cold start alone, and then some unholy amount of time to deploy the actual application that among thousands of other beans contains this single bean we need.

While this may have been true in the past, it’s indeed true that the current crop of application servers starts up in a second or less on modern hardware. Via tools such as Shrinkwrap it’s easy to define micro-deployments and deploy those with another tool like Arquillian in mere milliseconds.

Another much touted benefit of mocks is the ability to run tests independently of some global resource. For a large number of cases, this global resource is then the main database on which an application depends.

In this case we could ask ourselves whether this global resource is really a global one. Many databases can be installed locally with great ease (think PostgreSQL and MySQL). A best practice in software development is to be able to install the entire application stack that you’re developing on your local workstation anyway. Sure, there are exceptions if big external mainframes are involved, but for many common types of development this should be possible. If for some reason this is difficult or even impossible, then the team likely has much bigger problems to worry about. Namely, more often than not that difficulty is caused by things like software depending on “some server” that “someone once installed” and now “nobody knows what it exactly does”. This will not only make (unit) testing extraordinary difficult, but will also impede proper debugging and staging.

So, with some exceptions, could we perhaps say that mocks in this case are used to cover up another code smell? Surely this can’t be a good practice, can it?

A more profound statement uttered by the NoMock movement is that testing with mocks simply isn’t really useful. By extension they thus seem to be saying that unit tests are just not really useful.

Obviously, this is a controversial statement that goes against common wisdom, and it surely may shock people who always did unit testing “because it’s what we should do”, without giving it much thought. (will people who did think it through not be shocked, and will they simply revolt against such a statement?)

The canonical unit test example is that of the calculator. Here we test an add() method by providing it with some well chosen sample points. Two positive numbers, a negative and a positive number, two negative numbers, etc. Every time we check that the output is what we expected. Without a doubt unit tests are very helpful here.

But the simple add() method didn’t had any dependency that we needed to mock. It’s a pure “functional” thing; the output depends only on the input, and nothing else.

Let’s now look at an example that does use dependencies. Better yet, let’s dive in deep and go straight to an example where a collaboration between objects is the main goal. For that, consider a typical Service facade where a method starts a transaction and calls two DAOs that persist something to a DB. The idea is that both DAOs join the transaction, and that either the data from both DAOs ends up in the DB, or none at all. The DB has some contraints set for the data that we are persisting.

This is what the class looks like:

@Stateless
public class ServiceFacade {

    @EJB
    private DAO1 dao1;

    @EJB
    private DAO2 dao2;

    public void doStuff(SomeObject someObject) {
        dao1.foo(someObject);
        dao2.bar(someObject);
    }

}

Per the rules of the unit test, we now have to mock our two DAOs and then our test will swap in those mocks before calling the method that we want to test. In order to do that, we first need to add two setters for those mocks:

@Stateless
public class ServiceFacade {

    @EJB
    private DAO1 dao1;

    @EJB
    private DAO2 dao2;

    public void doStuff(SomeObject someObject) {
        dao1.foo(someObject);
        dao2.bar(someObject);
    }

    public void setDAO1(DAO1 dao1) {
        this.dao1 = dao1;
    }

     public void setDAO2(DAO2 dao2) {
        this.dao2 = dao2;
    }
}

Now we also need to create our mocks. Let’s create a mock that just remembers if it’s being called. A mock for DAO1 could look like the following:

public class MockDAO1 implements DAO1 {
    private boolean isCalled; // + getter

    public void foo(SomeObject someObject) {
        isCalled = true;
    }
}

The mock for DAO2 will look the same. Now we’re ready to do our test:

DAO1 dao1 = new MockDAO1();
DAO1 dao2 = new MockDAO2();
ServiceFacade serviceFacade = new ServiceFacade();
serviceFacade.setDAO1(dao1);
serviceFacade.setDAO2(dao2);

serviceFacade.doStuff(new SomeObject());

assertTrue(dao1.isCalled());
assertTrue(dao2.isCalled());

After we run this code, the test passes and we’re happy. The code works! YES! :D

Taking a step back; what did we *really* test? Well, uhm, we tested that Java was able to call two methods on two objects. Great, it’s able to do that.

What we however really wanted to test was whether the data actually ended up in our DB and whether both DAOs joined the transaction to see if the effects of DAO1 are correctly undone (rollbacked) when DAO2 throws. In our test with mock objects, we could build a mock DAO2 that throws an exception, but this will surely not make DAO1#isCalled false again.

Now we could try to build some mock transaction manager, perhaps store it in TLS, then make our mock objects “mock transactional” objects, and then we could… but who are we kidding here? What would we *actually* be testing?

Aren’t we not blindly following the rules of the unit test here, without giving it a second thought of why we are actually doing it and what benefit it brings us? It’s this a bit like putting auto-generated comments above our methods, just to make the code analyzer happy?

Maybe those NoMock guys got me to rethink my strategy…

About these ads
  1. Royi Hagigi
    September 11, 2012 at 10:38 pm

    Maybe you should be testing DAO1 and DAO2 for those conditions. The test for doStuff is just supposed to verify that doStuff does the “stuff” it is supposed to do. You verified that it does do that.

  2. September 12, 2012 at 12:44 am

    Awesome! You explained things much better than I did.

  3. Chris F
    September 12, 2012 at 3:32 am

    The example given is a poor candidate for unit testing. I agree with you that there isn’t much point in testing doStuff, since all the work is done in the dependencies. As is, there is no behavior in ServiceFacade that is worth writing a unit test for. Since you mentioned interest in the database, if anything this is a case for a test where you would set up a local or in-memory db to test on. That’s not really a unit test, but it’s still useful.

    I’m a big fan of unit testing and write tests where it makes sense. However there are plenty of scenarios where other types of tests are more productive.

    Likewise, mocking has its place, but I think only after the method under test has been isolated as much as possible. This means separating code that does unit testable Stuff from the code that hits a db/file/webservice/whatever. Plenty of mocking libraries make mocks very easy to create, but I’d rather maintain unit tests that test isolated logic as opposed to having to inject a bunch of mock dependencies.

    Of course it’s important to avoid blindly following practices just “because you should” whether unit testing, mocking, or whatever.

  4. agoncal
    September 12, 2012 at 7:38 am

    Can’t agree more with that. This is what I wrote a few months ago : http://antoniogoncalves.org/2012/01/16/wytiwyr-what-you-test-is-what-you-run/

  5. zilluss
    September 12, 2012 at 8:04 am

    What you’re looking for are integration tests. In those tests you don’t mock the dependencies and you actually hit the database (See http://schneide.wordpress.com/2011/09/05/a-shot-at-definitions-beyond-unit-test/ for a nice and comprehensible distinction of the various test types).

    I recommend you to have a look at the book “Growing Object-Oriented Software, Guided by Tests” (No, I don’t get any money for ads, the book is really good!) which covers all the topics you addressed (testing the db, testing correct communication among dependencies) and more (testing the UI)

    TL;DR: Don’t confuse integration with unit tests.

  6. September 12, 2012 at 2:58 pm

    The article seems straight forward, but misses the point of testing against mock objects… in your example, you are correct… you are simply testing whether two functions were called… in objects that will never be used in production… entirely a waste…

    instead, how about a mock object that INTENTIONALLY FAILS?… test whether the other object fails as well… are the methods called that will roll back the transaction?

    I would suggest looking into BDD style testing, which is more suitable for integration tests anyway… then also consider what you’re testing and whether you have value in the test… DAL tests SHOULD be hitting against live databases… but those should also be specific, like writing valid data, or rolling back a transaction.

    The problem (which I’ve seen/heard/read about more times than I can count) is that people seem to like writing tests for the sake of writing tests… tests need to have value… otherwise you are correct, you’re just wasting time.

    • September 12, 2012 at 6:42 pm

      Instead, how about a mock object that INTENTIONALLY FAILS?

      That’s a good one, although the object I would use for that perhaps isn’t exactly the traditional mock and the setup also wouldn’t be the plain unit test one.

      In that case, the “mock” would be an EJB as well, but indeed one that immediately throws an exception. The test would then be run in an embedded or external container (most likely via Arquillian).

      I might be wrong, but I’m afraid that the strict unit test “followers” would already consider that an integration test. At the end of the day, I don’t care so much if the tests that I run are technically considered to be integration tests or unit tests. I just want them to be useful.

      Another way to let the test intentionally fail is by loading the (test) database with data that’s known to cause a conflict. For instance, intentionally having a row to which an entity is mapped with an ID that’s one higher than the highest value of the sequence used to generate IDs. A newly persisted entity will then cause a duplicate PK exception of some sorts .

      tests need to have value… otherwise you are correct, you’re just wasting time.

      Yes, that’s indeed main point ;)

  7. September 12, 2012 at 6:05 pm

    Fantastic writeup, and I like the simplicity of the “NoMock Movement” moniker. For the past few years our techs, docs and talks have centered around the notion that when you use a component model (Java EE, @Inject, etc), you’re relying heavily on software you didn’t write to wire up your code paths.

    An EJB is only an EJB inside an EJB container. A Servlet is only a Servlet in a Servlet container. If you use the “new” operator, you’re testing a POJO, not an enterprise component.

    Arquillian is the component model which makes true integration testing look and feel like unit testing.

    • September 12, 2012 at 6:49 pm

      An EJB is only an EJB inside an EJB container. A Servlet is only a Servlet in a Servlet container. If you use the “new” operator, you’re testing a POJO, not an enterprise component.

      Very well said!

      In a way the container can be seen as a kind of extension to the JVM. Thinking of it, who would test a plain Java Object outside of a JVM? Nobody would say that’s a good idea, so why is it then that some people think testing an EJB, Servlet, CDI bean etc created by “new” is a good idea?

  8. Dainius
    September 13, 2012 at 2:34 pm

    If you want test small piece of code you write unit test, that’s why it’s called unit test in first place. And that’s why you need moc objects to avoid influence from outside.
    If you are testing how two components works, use integration test and call it integration test, don’t call it grape if it’s pear.
    But that’s IMO general problem, that most developer don’t recognize difference, and unit testing is so popular name, that it’s very easy to call all test – unit test.

  9. marciopd
    September 14, 2012 at 5:32 pm

    Maybe it’s just wrong to write unit tests and expect the benefits of integration testing.

    People are really get confused about this.

    Units tests improve a lot the codebase. Small bugs are quickly found and complex code claims for refactoring.

    You can unit test your daos using dbunit.

  10. October 2, 2012 at 12:38 pm

    http://www.jbrains.ca/series/integrated-tests-are-a-scam

    what a show it would be to see you both doing a sample project in course of some tech days (or webcast). The approaches are tempting but I still can’t decide for myself what is better.

  11. October 2, 2012 at 12:42 pm
  12. jbrains
    November 19, 2012 at 5:20 pm

    A few short comments.

    First, a purely abstract example will not illustrate the benefits of programming to abstractions (for me, the key reason for mock objects), because there are no abstractions to find when we don’t know the context. With a real example, testing objects in isolation, and the tension that comes with it, throws new abstractions in front of my face. Much more often than not, it simplifies my design, and gives me opportunities to either extract reusable code or replace custom code with a trusted library.

    Second, your example shows mocking entities, rather than services, which I tend not to do. Reference: http://link.jbrains.ca/WpR9aS

    Third, with your example, you claim that all you’ve tested is that Java can invoke two methods in a row. I don’t see it that way. You’ve tested that ServiceFacade enforces that foo() happens before bar(). In a specific context, that might matter a great deal, if foo() is authenticate() and bar() is updateSensitiveResource(). Given how important it is to do that in the proper sequence, I’d love to have that code in one place and one place only, and to know that it works.

    Finally, I agree with @marciopd: you can’t expect the benefits of an integrated test from an isolated test. To me, the idea “rollback when catching an exception” is a reusable bit that I want available everywhere in my system, so I implement that as a Template Method that delegates the body of the work to an interface. This makes it easy to test the Template Method without worrying about the “unit of work”, and you can choose whether to execute the unit of work in the transaction context or not. This is the Context Independence principle that Freeman and Pryce write about in “Growing Object-Oriented Systems”. Reference: https://gist.github.com/4112020

    Mocks or not is a style choice. I don’t mind either way. Blaming the mocks for complicated tests is like blaming the crunches for the pain you feel in your fat stomach. Blame the fat, instead.

    • November 21, 2012 at 3:26 pm

      Second, your example shows mocking entities, rather than services

      The DAO is actually a specific type of service. It’s definitely not an entity. SomeObject in my example would correspond to an entity. If however the concept of a DAO is what bothers you here, just read “Service” where I use “DAO”. The intent was to use services or service-like classes here.

      You’ve tested that ServiceFacade enforces that foo() happens before bar(). In a specific context, that might matter a great deal,

      I agree, in some contexts that indeed matters. Not so here. In my example the order does not matter. Both operations are being done atomically in 1 transaction. Barring any unwanted visibility (transaction isolation) concerns, the order definitely can’t matter here. I’m not entirely sure, but I think 2PC for distributed transactions doesn’t give any guarantees about the order in which resources that participate in a transaction are committed. It just says that all of them need to be prepared and that it gives all of them the commit command.

      I think it would even be impossible to guarantee that changes made to the transactional resources actually become generally visible in order.

      so I implement that as a Template Method that delegates the body of the work to an interface

      Uhm, just a question, but do you know how EJBs and the EJB container works? This is basically what the EJB container does, and why it’s so important to test in-container (using mocks or not).

      • jbrains
        October 6, 2013 at 8:24 pm

        > Uhm, just a question, but do you know how EJBs and the EJB container works?

        Yes. I admit, my knowledge is based on EJB 2, so I remain prone to some old-style thinking, and more recent EJB specifications might have improved the situation. Moreover, just because the EJB container behaves this way doesn’t alone make this pattern less useful.

        I suppose it’s possible that changes in the EJB specification have outdated many of my opinions, but the entire EJB chapter of my book JUnit Recipes outlines some arguments when to test in-container and when not. While I don’t use EJB daily anymore, I get to talk about it every so often with programmers who do, and from what they’ve told me, the essential problem remains: programmers write EJBs far too aware of their runtime context, which forces them to deploy those components in order to run even that behavior which could run entirely in memory. They’ve missed the benefits of context independence completely, and it doesn’t take much: a single annotation that assumes the presence of the EJB container and you’re out of luck.

  13. July 15, 2013 at 8:32 pm

    Yeah – I agree with a couple of the replies: there’s a difference between integration tests and unit tests – expecting that we get functionality testing from a unit test is a bad expectation. The purpose of unit testing goes beyond testing if functionality works – it tests that your contracts remain the same between the micro-parts of your system. And EVEN MORE important – it guides your design so that your objects don’t become inappropriately coupled. For a fantastic coverage of this topic, see: http://www.growing-object-oriented-software.com/

    If you let your tests guide your design and implementation – you write code significantly faster. Which is the ultimate promise….

  1. September 17, 2012 at 4:41 pm
  2. November 27, 2012 at 2:56 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: