So far all the extension testing that I have been doing were manual tests; I defined a procedure, followed it, and noted down the discrepancies between the expected and the actual behaviour. For certain extensions this is the only feasible way, but when a given input creates a given output this whole process can be automated.Wikiation is developing a testing environment that works in a similar way to the MediaWiki core parser test system but it is more flexible. It seemed a good moment to test the Cite extension. As I am now able to use automated tests, I can repeat this more often and be much more efficient.
Improved reliability and accuracy is why you want to use automated tests. It prevents human errors and doing the same boring stuff is something computers are good at. Making sure that new releases of MediaWiki work with the installed extensions is a lot easier. All the tests can be run again and again, so when things continue to work like they used to, there is a good indication that everything is fine.
Posts
2 comments:
Sweet stuff, MinuteElectron. Care to elaborate on how this will be further automated? I can think of daily tests on trunk, uploading results somewhere, testing branch and trunk extension version against other version, updating the matrix on for extension version compatibility on MediaWiki.org. Adding the test framework files to svn.wikimedia.org, so that the format gets some exposure, hopefully leading to extension developers committing the unit tests together with their extension work, etc.
Siebrand this is only the start. The goal is that all extension user and builders can always test their test cases as pre-defined by the extension
user/builder. You can test as often as youlike. Define the test cases, include the defined results for your tests and compare the different versions of an extension. It is possible to run the same extension with the same test case on different servers with different server configurations. This will make sure if it works on your server configuration as well as on the server of the developer.
There is a lot of work to be done and I am happy you like the Wikiation Testing Environment. I share your appreciation for the work done by MinuteElectron.
Post a Comment