I spent the past 11 months (as I mentioned previously) as part of a team of developers on an 'agile' project, using the Extreme Programming methodology. As I was working out today, I was reading the December 2004 issue of asp.net Pro magazine. The featured article is about Test Driven Development (TDD), and it's place in agile development.

First, let me preface this topic by saying I am absolutely FOR Test Driven Development. In my opinion, there's too little focus given by developers on focusing in what can go wrong with an application -- testing, error handling, malicious users, and a myriad of other potential gotchas. The purpose of this blog is simply to provide some alternative views on TDD, and some potential pitfalls, since there is little seen on the topic, especially from those of us in the trenches. Combine this in your approach to TDD, and you'll have a much better experience.

Here's an executive summery of the caveats to TDD from my experience:

1. No code is supposed to be written before the tests. Design first, then write the tests, or you'll end up with a bad design.
2. When tests take longer to write and modify than the actual coding, the process gets in the way.
3. TDD gives way to the erroneous philosophy that if your automated tests work, that's all the testing you need (except user acceptance).
4. There is the dangerous thinking that you can write automated tests to cover every scenario. See point 2 again.
5. There becomes a belief that the number of tests somehow correlates to the quality of testing, or the the mere existence of an automated test provides a better application.
6. UI testing tools (especially the open source varieties) are lacking in the ability to generate UI tests, at least from the web perspective in the trench I sat in.
7. When you data shunt the data access (and manipulation) portion of a class, that's not a very valid test. The database MUST be tested, and your tests must accurately reflect how the users are going to be actually using your objects for TDD to be of any use.

Let's look at each point in a bit of detail, so I can provide some details on my OPINIONS here...

First, the statement is made no code should be written before the test. OK, what exactly are you going to be testing for? For example, let's say you might be wanting to test customer creation. You envision the routine you want to test to return an integer which ties to the ID the database generates. OK, great! But guess what, that's a DESIGN DECISION. Why not set a read-only property of a class, and return a standard return code? Are you going to follow this exact style with the rest of your classes? Will the other developers do the same? Will the routine throw an error, or handle it itself? If handled, how will I know there was an error?

I know for a fact that if you don't lay out some very specific coding guidelines and design, four developers are going to design the same class four different ways, even when pairing. Given that fact, how can you possible write meaningful and consistent tests without doing a basic architecture/design first.

Second, writing tests is time consuming. Maintaining tests can be even worse. On this same project, test maintenance at times took more time than making a simple change. Why? When the change is made, many times a refactoring occurs (in our case it was almost a given since there was little thought placed on a unified design -- some day I will write on why that occurred as part of a human dynamics of development topic). Suddenly, the test needs to be re-written because the test now fails as a result of the refactoring. A simple business rule change has now turned into a design and test change. A task that might have taken an hour now takes two days. Sorry, as a business owner, it's a bitter pill to swallow.

Another mistake occurs when the developer takes on the attitude that when the automated tests pass, the testing is complete. Let's face it, as developers, testing sits right up there with documenting code. No one wants to do it, but it HAS to be done. With TDD, it's an easy trap to fall into. I even heard “My tests run, let the customer catch anything else in User Acceptance.” Wrong! The customer's are why we as developers have jobs. We need to do due diligence in functional testing, integration testing as well as usability testing. Automated testing is NOT going to do that for us. To think your automated tests are going to catch all the errors is dangerous thinking. No one has every promoted TDD to accomplish this, but I can tell you from experience, the possibility that this mindset can exist is there.

As I mentioned before, I am all for TDD and automated testing. However, I also like to be thorough. I also only like to write valid tests. If I was to sit there and ponder every single possible scenario, at my bill rate the client is going to be looking elsewhere thinking I am nor getting enough done. Consultants beware! But seriously, even if I could take the time to come up with test which can adequately test the possible scenarios, how much have I gained (in the short term -- long term is a whole other story which TDD clearly wins)? Testing scripts which repeatedly test the application, written by the users and repeated each time in user acceptance is a far more effective testing methodology (OK, devil's advocate time -- can you really get them to test it completely every time? The tester also suffers from laziness or lack of attention in repetitive tasks..) So writing tests is going to also be an iterative process which never ends. When a bug shows up, we need to refactor our testing to cover that scenario in the future. Now I am maintaining two projects which is even more time consuming.

The next fallacy in TDD is that the sheer number of tests (metrics) means you have good testing. In the environment I was working in, their reviews were actually based in part on the number of tests they wrote. Never mind that the majority of the test written were actually an effective test, but the sheer count was taken to be effective testing. Quality in testing should be your number one priority. If you can only come up with one test which effectively tests an object, then so be it. Adding five more test which do nothing other than add time during a refactoring is little help to the cause of good software.

Right now, automated testing really focuses on the Data Access Layer (DAL) and Business Objects Layer (BOL). When I discuss shunting next, you'll see it actually comes down to the BOL in many cases. What about the UI? The tools, especially for ASP.Net in the open source community are severely lacking in automated testing. How will you know if the javascript works both on Netscape and IE without running it yourself?  It's the UI the user sees, which is why the UI has to be thoroughly tested in both how you expect the user to use it, and how they might use it. Users always come up with useful scenarios to break our applications. Think of it as a fun challenge and you'll be far less frustrated.

To test out our BOL, we shunted the data using XML or pre-defined data tables. It actually works great, but it's a limited test. What happens when the database schema changes? If you've shunted the data, you'll never know the impact unless you do your due diligence and fully integrate test, or the user finds it in user acceptance testing (which is not where you want to catch these things). If you do shunt your data, that's great and fine. Just make sure you test against the schema the application is going to be deployed against as well. See my blog about getting a random record using NewId() if you want to generate some good database based tests.

OK, it probably sounds like I am really bummed about automated testing after my recent experience with it. That belief is the farthest from the truth. In fact, I am going to introduce it on my current project because I feel it can certainly add a key dynamic to a complete testing process. But as Test Driven Development gains acceptance, I felt it was time someone at least mentioned the things to consider so it can be a more effective tool in your development cycle. Plan ahead and determine how you can avoid these pitfalls and hopefully your investment in time will generate more dividends from Test Driven Development.