(This post was originally published on the INFORMS 2013 Blog).
While attending this year’s INFORMS Annual Conference, this phrase caught my attention: we need to keep a horse in the race. It simply means that for a typical paper to be accepted in a journal, researchers either need a completely new model or application, or they need to demonstrate the superiority of their proposed model / algorithm. In order to do the later, authors need another approach to compare with – that is, keeping a horse in the race that they can beat.
The interesting paradox is: nobody wants to beat a lame horse. Reviewers are especially wary of obviously lame horses. On the contrary, in order to prove the validity of one’s method, it is advised to compare against the “best” available. In mixed-integer programming literature, this comparison is usually made against either CPLEX or Gurobi. The irony in that is, the more one solver gets “beaten” by specific algorithms in the literature, the more it helps establish that solver’s reputation as “the thing to beat”.
In the metaheuristics / AI community, one usually compares either with the method that did score best in the literature (meaning that this one should become the state-of-the-art) or with a more basic implementation of that algorithm (showing that additional complexity pays off).
That being said, academia have honor. Every researcher will tell you how hard it is often to beat the generic solvers, both commercial and open-source. Furthermore, they will also tell you that the horse in question always keeps improving, and at a pretty quick pace.
Some horse selection tips
It is advised to pick an application / algorithm that is:
- Widely available and used
- A recent version of it, hopefully still maintained
- Easy to understand / to use
- Has been cited or used as a horse in the relevant literature.