Introducing Testinium (v 1.0)
What is Testinium?

Testinium is a pet-project of mine. Its goal was to be used as a replacement of JUnit for J2SE 5.0 applications. I never published the source code, until now. Once I became aware of TestNG, I stopped using my version because I had not developed an Eclipse plug-in (although planned), and TestNG provided a plugin.
Testinium was never meant to be as complete as what TestNG provides, or what JUnit 4 will provide, but let's see what I can do with my free time.

I have decided to walk you through some stuff I have done for Testinium. Consider this as a tutorial of writing your own Testing Framework...(YATF).

Thierry Janaudy

License > Testinium is released under Apache License Version 2.0
How do I write a Unit Testing framework?

The choice is yours, but when I started to work on Testinium, I did not have to worry about JUnit, legacy-code or any other things. I just focused on J2SE 5.0.
NUnit and TestNG introduced the concepts of attributes (or annotations) for testing purposes. TestNG uses the @Test annotation to annotate methods that need to be tested, with some other stuff like number of methods invocations, groups and dependencies,...

Testinium defines a @Testable annotation.
Dependencies

One of the great features of TestNG is the notion of groups and dependencies between groups and methods. When re-building Testinium, I decided to keep that feature using the @Group annotation.
Testinium has the following characteristics:

> A @Testable method with no @Group belongs to "Default Group"
> A @Testable method can depend on zero or more @Testable methods
> A @Group can depend on zero or more @Group groups
Annotations

With the previous elements in mind, we can start to design the annotations :		 

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface Testable {
 /**
  * dependsOn
  * @return the dependend methods this method depends on
  */
public String[] dependsOn() default {};
}
Testable.java 

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface Group {
 /**
  * name
  * @return Name of this group
  */
public String name();
	
 /**
  * dependsOn
  * @return The groups this group depends on
  */
public String[] dependsOn() default {};
}
Group.java
A basic use of those annotations would be : 

package org.jyperion.testinium.samples;

import org.testinium.annotations.Group;
import org.testinium.annotations.Testable;

 /**
  * BasicTest
  * 
  * @author Thierry Janaudy janaudy@testinium.org
  * @version 1.0
  */
public class BasicTest {
@Testable
public void m1() {
System.out.println("m1");
 }
	
@Testable(dependsOn = {"m1"})
@Group(names = {"GroupA"})
public void m2() {
System.out.println("m2");
 }
	
@Testable
@Group(names = {"GroupB", "GroupC"}, dependsOn = {"GroupA"})
public void m3() {
System.out.println("m3");
 }
}
BasicTest.java
Sorting out dependencies

The next step in writing a Unit Testing framework is to work out the dependencies, what method should be invoked first, and detect cycles if any. You can use a mix of Depth Search First algorithm with some Topological sorting.

The DFS algo is :		 

DFS(G)
for each vertex u in Graph
  do state[u] <- UNVISITED
time <- 0
for each vertex u in Graph
  do if state[u] = UNVISITED
       then Visit(u)

Visit(u)
state[u] <- VISITING
d[u] <- ++time
for each v in successors[u]
  do if state[v] = UNVISITED
    Visit(v)
state[u] <- VISITED
f[u] <- ++time
DFS Algorithm
The previous DFS algorithm can be enhanced to detect cycles : 

DFS(G)
if all vertices have a parent, then cycle detected
for each vertex u in Graph
  do state[u] <- UNVISITED
time <- 0
for each vertex u in Graph
  do if state[u] = UNVISITED
       then Visit(u)

Visit(u)
state[u] <- VISITING
d[u] <- ++time
for each v in successors[u]
  do if state[v] = VISITING and v is NOT a predecessor of u
       cycle detected
     if state[v] = UNVISITED
       Visit(v)
state[u] <- VISITED
f[u] <- ++time
DFS Algorithm with Cycle Detection
Once nodes have been VISITED, they are added to a list, forming a topologically sorted list.
Working with classes and instances

If Testinium is given a .class, it can check for annotations using the method getAnnotations() from java.lang.Class.
However, you may want to be a little bit more flexible than this. Because there is no notion of annotation inheritance on the Java platform, what if you annotate a Remote interface (like an EJB interface) using @Testable? Nothing if you just give the instance (because you might get a proxy, or an inherited class), and nothing if you just give the .class.

Testinium supports the following :

.class > Testinium uses newInstance() to get the instance (hence you must provide a zero-argument constructor)
instance > Testinium uses getClass() to get the Class object
.class and instance > Testinium uses the .class to get the annotations and the instance to invoke the @Testable methods

ClassInfo.java is the class holding that information :		 

public class ClassInfo {
 /**
  * Class
  */
private Class<?> clazz = null;
	
 /**
  * instance
  */
private Object instance = null;
	
 /**
  * ClassInfo
  * @param clazz
  * @throws IllegalAccessException 
  * @throws InstantiationException 
  */
public ClassInfo(Class<?> clazz) throws InstantiationException, IllegalAccessException {
     this.clazz = clazz;
     this.instance = this.clazz.newInstance();
 }
	
 /**
  * ClassInfo
  * @param clazz
  * @param instance
  */
public ClassInfo(Class clazz, Object instance) {
     this.clazz = clazz;
     this.instance = instance;
 }
	
 /**
  * ClassInfo
  * @param instance
  */
public ClassInfo(Object instance) {
     this.instance = instance;
     this.clazz = this.instance.getClass();
 }
}
ClassInfo.java
Have you seen an annotation somewhere?

The next step is to look for annotations in the classes provided to Testinium. The @Testable/@Group methods are placed into a graph. A graph has nodes, a node may have children and/or parents. Each node has a containee, the node being the container.
For each class that Testinium processes, we have to go through all the annotated methods and build the graph.

> If a method is annotated with @Group only, a RuntimeException is thrown
> If a method is annotated with @Testable only, it is placed in a group called "Default Group"
> If a method has dependencies on other methods, this method becomes the parent node of all the other dependent methods
> If a group has dependencies on other groups, this method becomes the parent node of all the other dependent methods
Processing the classes

The next big thing is to process the classes and build the non-yet-sorted graph. This is done by the process() method in ClassInfo.java
(Could be optimized - will be refactored in new versions of Testinium) : 


/**
 * process
 * @throws NoSuchMethodException 
 * @throws SecurityException 
 * @throws CycleException 
 */
public void process() throws SecurityException, NoSuchMethodException, CycleException {
  for(Method declaredMethod : this.clazz.getDeclaredMethods()) {
    if (declaredMethod.getParameterTypes().length > 0 || // Ignoring methods with arguments
        declaredMethod.getAnnotations().length == 0) // Ignoring methods with no annotations
      continue;
			
    if (declaredMethod.getAnnotation(Group.class) != null &&
        declaredMethod.getAnnotation(Testable.class) == null)
      throw new RuntimeException("Method " + declaredMethod.getName() + " must define @Testable");
			
    Testable testable = null;
    if ((testable = declaredMethod.getAnnotation(Testable.class)) != null) { // @Testable is defined
      MethodInfo declaredMethodInfo = new MethodInfo(declaredMethod);
      INode declaredMethodNode = null;
      MethodNodeKey declaredMethodNodeKey = new MethodNodeKey(declaredMethod.getName()); 
      if (!this.graph.getNodesMap().containsKey(declaredMethodNodeKey)) {
        declaredMethodNode = new NodeImpl(declaredMethodInfo);
        this.graph.getNodesMap().put(declaredMethodNodeKey, declaredMethodNode);
      } else
        declaredMethodNode = this.graph.getNodesMap().get(declaredMethodNodeKey);
				
      for(String dependentMethodStr : testable.dependsOn()) {
        StrNodeKey dependentMethodStrNodeKey = new StrNodeKey(dependentMethodStr);
        INode dependentMethodNode = this.graph.getNodesMap().get(dependentMethodStrNodeKey);
        if (dependentMethodNode == null) {
          Method dependentMethod = this.clazz.getDeclaredMethod(dependentMethodStr, new Class[]{});
          if (dependentMethod.getAnnotation(Testable.class) == null)
            throw new RuntimeException("Dependent method " + dependentMethodStr + " does not define @Testable ");
          MethodInfo dependentMethodInfo = new MethodInfo(dependentMethod);
          dependentMethodNode = new NodeImpl(dependentMethodInfo);
          this.graph.getNodesMap().put(dependentMethodStrNodeKey, dependentMethodNode);
        }
        declaredMethodNode.successors().add(dependentMethodNode);
        dependentMethodNode.predecessors().add(declaredMethodNode);
      } // for(String dependentMethodStr
				
      Group group = declaredMethod.getAnnotation(Group.class);
      if (group == null) {
        GroupNodeKey groupNodeKey = new GroupNodeKey(DEFAULT_GROUP);
        INode groupNode = this.graph.getNodesMap().get(groupNodeKey);
        if (groupNode == null) {
          groupNode = new NodeImpl(DEFAULT_GROUP);
          this.graph.getNodesMap().put(groupNodeKey, groupNode);
        }
        groupNode.successors().add(declaredMethodNode);
        declaredMethodNode.predecessors().add(groupNode);
      } else {
        for(String groupNameStr : group.names()) {
          GroupNodeKey groupNodeKey = new GroupNodeKey(groupNameStr);
          INode groupNode = this.graph.getNodesMap().get(groupNodeKey);
          if (groupNode == null) {
            groupNode = new NodeImpl(groupNameStr);
            this.graph.getNodesMap().put(groupNodeKey, groupNode);
          }
          groupNode.successors().add(declaredMethodNode);
          declaredMethodNode.predecessors().add(groupNode);
          
          for(String dependentGroupStr : group.dependsOn()) {
            GroupNodeKey dependentGroupNodeKey = new GroupNodeKey(dependentGroupStr);
            INode dependentGroupNode = this.graph.getNodesMap().get(dependentGroupNodeKey);
            if (dependentGroupNode == null) {
              dependentGroupNode = new NodeImpl(dependentGroupStr);
              this.graph.getNodesMap().put(dependentGroupNodeKey, dependentGroupNode);
            }
            groupNode.successors().add(dependentGroupNode);
            dependentGroupNode.predecessors().add(groupNode);
          }
        } // for(String groupNameStr
      } // else
    }
  } // for(Method declaredMethod
		
  this.graph.prettyPrint();
  this.invokeTests();
} // process
process()
Calling DFS on BasicTest.java

The following list is the topologically sorted nodes list for BasicTest.java :

[Testinium] version 1.0
Node Default Group
 -> MethodInfo: m1
-------------------------
Node GroupA
 <- GroupB <- GroupC
 -> MethodInfo: m2
-------------------------
Node MethodInfo: m1
 <- Default Group <- MethodInfo: m2
-------------------------
Node MethodInfo: m3
 <- GroupB <- GroupC
-------------------------
Node GroupB
 -> MethodInfo: m3 -> GroupA
-------------------------
Node GroupC
 -> MethodInfo: m3 -> GroupA
-------------------------
Node MethodInfo: m2
 <- GroupA
 -> MethodInfo: m1
-------------------------

MethodInfo: m1 <- Default Group <- MethodInfo: m2 <- GroupA <- MethodInfo: m3 <- GroupB <- GroupC
You should read the last bit <- as "dependent on" :

> GroupC depends on GroupB
> GroupB depends on m3
> m3 depends on GroupA
> GroupA depends on m2
> m2 depends on Default Group
> Default Group depends on m1

And the methods will be invoked in the following order :

> m1, m2, m3
Checking on a failed test

Testinium supports basic exceptions and the assert keyword from J2SE 5.0. To mark a test as a failed test, you must use the assert keyword or throw an exception. If throwing an exception is part of the expected behaviour of the test method, it should be declared using the @ExpectableExceptions annotation 

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface ExpectableExceptions {
 /**
  * Exceptions that may be thrown
  * @return classes
  */
public Class[] classes();
}
ExpectableExceptions.java
Invoking the test methods

Methods are called by the invokeTests() method in ClassInfo.java: 


public List<INode> invokeTests() throws CycleException {
  List<INode> sortedList = DFS.topoSort(this.graph.getNodesMap().values());
  for(INode sortedNode : sortedList) {
    Object containee = sortedNode.getContainee();
    if (containee instanceof MethodInfo) {
      MethodInfo methodInfo = (MethodInfo)containee;
      Method method2test = methodInfo.getTestableMethod();
      try {
        method2test.invoke(this.instance, new Object[]{});
        methodInfo.setFailed(false);
      } catch (Throwable throwable) {
        methodInfo.setFailed(true);
        StringWriter stringWriter = new StringWriter();
        PrintWriter printWriter = new PrintWriter(stringWriter);
        throwable.printStackTrace(printWriter);
        methodInfo.setStackTrace(stringWriter.toString());
					
        ExpectableExceptions exceptions = null;
        if ((exceptions = method2test.getAnnotation(ExpectableExceptions.class)) != null) {
          for(Class clazz : exceptions.classes()) {
            try {
              Object castedObject = clazz.cast(throwable.getCause());
              methodInfo.setFailed(false);
            } catch (ClassCastException classCastException) {}
          }
        } else {
          methodInfo.setFailed(true);
        }
      }
    }
  } // for(INode sortedNode
		
  return sortedList;
} // invokeTests
invokeTests()
That's it! We have now a basic testing framework on top of which we can build. Note that this is not intended for production.
Let's see how we can extend Testinium to support other features.
Generating reports

In order to receive events on tested methods, implement the MethodInvokerListener interface: 


public interface MethodInvokerListener {
  /**
   * onBegin
   */
  public void onBegin();
	
  /**
   * onTestedMethod
   * @param className
   * @param methodName
   * @param groupName
   * @param failed
   * @param stackTrace
   * @param milliTime
   */
  public void onTestedMethod(String className, String methodName, String groupName, boolean failed, String stackTrace, long milliTime);
	
  /**
   * onEnd
   */
  public void onEnd();
}
MethodInvokerListener.java
.. and register it via: 


Main testinium = new Main();
testinium.addListener(new ConsoleReporter());
testinium.addListener
I may add this in a simple config file. But I won't bother because you'll get the source code to DYI.

Testinium is provided with two listeners: a ConsoleReporter and a HTMLReporter
Number of invocations

TestNG introduced the invocationCount attribute. It is a very neat way to tell the testing engine to invoke the test method more than once. You may use this to measure performance on a method call.
Let's add an attribute numberOfInvocations to the @Testable annotation: 


@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface Testable {
  /**
   * dependsOn
   * @return the dependend methods this method depends on
   */
  public String[] dependsOn() default {};
	
  /**
   * numberOfInvocations
   * @return 1 by default
   */
  public int numberOfInvocations() default 1;
}
Amended Testable.java
If you choose numberOfInvocations > 1, what should happen to the time measured? If one test fails but nine tests succeed, do you mark this test as failed? TestNG solves this by using a successPercentage() attribute. I will add support for this in version 2.0. For now, Testinium 1.0 will mark the test as a failed test if there is at least one failure. The time is the average invocation time.
The result is shown in this HTML Report.
Parallel testing and timeouts

The most enjoyable feature of TestNG is its support for parallel testing and timeout. We need some thinking here to add that feature to Testinium.
The Concurrency Utilities provide everything we need to add support for parallel testing and timeouts to Testinium.

In order to implement parallel testing, we can use ThreadPoolExecutor that is part of the Concurrency Utilities of J2SE 5.0.

The idea is to have a pool of threads (with one active thread by default) that execute FutureTasks. The number of threads would need to be configured at the class level (We need another annotation at the class level). Each FutureTask has the public V get(long timeout, TimeUnit unit) method that can be used to figure out if a timeout has expired before the method has completed.

But first, let's define a @Configuration annotation that could be used at the class level to define the parallel strategy.
Configuring the parallel mode
 

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target(TYPE)
public @interface Configuration {
  /**
   * Should the methods be run in parallel mode or not?
   * @return false by default
   */
  public boolean parallel() default false;
	
  /**
   * Number of threads in the pool
   * @return 1 by default
   */
  public int numberOfThreads() default 1;
}
Configuration.java
A sample class would look like this: 


@Configuration(parallel = true, numberOfThreads = 10)
public class ParallelTest {
  @Testable
  public void test1() {
  }
}
ParallelTest.java
Note that if parallel is set to false, it does not matter how many threads you have, Testinium will use one (1) thread. We also need to amend the @Testable annotation with a timeout attribute: 


@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD})
public @interface Testable {
  /**
   * dependsOn
   * @return the dependend methods this method depends on
   */
  public String[] dependsOn() default {};
	
  /**
   * numberOfInvocations
   * @return 1 by default
   */
  public int numberOfInvocations() default 1;
	
  /**
   * Specifies how much time Testinium should wait before 
   * @return
   */
  public long timeOut() default Long.MAX_VALUE;
}
Amended Testable.java
Result

With all of this in place, let's test the following class: 


@Configuration(parallel = true, numberOfThreads = 10)
public class ParallelTest {
  @Testable(timeOut = 100)
  public void test1() throws InterruptedException {
    Thread.sleep(200);
  }

  @Testable(timeOut = 200)
  @ExpectableExceptions(classes = {InterruptedException.class})
  public void test2() throws InterruptedException {
    Thread.sleep(400);
  }

  @Testable(timeOut = 300)
  public void test3() throws InterruptedException {
    Thread.sleep(200);
  }

  @Testable(timeOut = 400)
  public void test4() throws InterruptedException {
    Thread.sleep(400);
  }

  @Testable(timeOut = 500)
  public void test5() throws InterruptedException {
    Thread.sleep(400);
    assert 400 == 500;
  }
}
ParallelTest.java
Testinium generates the following HTML report.

What's next?

It took me about 2 days to write Testinium 1.0. Most of the ideas come from TestNG, so if you want a piece of advice, use TestNG.
I would like to work on Testinium 2.0, a re-write and refactored version with the following features:

> A plug-in architecture to allow your own sorting algorithm for example. You would be given a graph, and you would return a sorted list of methods to invoke, and so on
> Refactored code, because j'ai développé ça comme un cochon
> Develop an Eclipse plug-in
> Develop a NetBeans plug-in
> Add TestNG tests to test Testinium :-)
> Add Testinium tests to test Testinium and compare them with TestNG
> Fix the numerous bugs that I have not yet found (I am not pleased with my use of ThreadPoolExecutor)
> An ANT task

Also, Testinium will always target J2SE 5.0 and beyond. Testinium will never support JUnit.