Automated Extension Testing

So far all the extension testing that I have been doing were manual tests; I defined a procedure, followed it, and noted down the discrepancies between the expected and the actual behaviour. For certain extensions this is the only feasible way, but when a given input creates a given output this whole process can be automated.

Wikiation is developing a testing environment that works in a similar way to the MediaWiki core parser test system but it is more flexible. It seemed a good moment to test the Cite extension. As I am now able to use automated tests, I can repeat this more often and be much more efficient.

Improved reliability and accuracy is why you want to use automated tests. It prevents human errors and doing the same boring stuff is something computers are good at. Making sure that new releases of MediaWiki work with the installed extensions is a lot easier. All the tests can be run again and again, so when things continue to work like they used to, there is a good indication that everything is fine.

The Configure Extension and Its Relevance

Recently I have started testing the Configure extension, it is very valuable to our project for a variety of reasons - and indeed quite unlike most of the other extensions we test.

Primarily it will help to ensure that testing is more comprehensive, if extensions are supported by configure then it reduces the chance that a software tester (such as myself) will miss out testing certain configuration settings.  Making sure that all configuration settings are tested at least independently, and even better together (of course, this isn't always feasible), is integral to the validity of results. Working by both reading the settings in the extension's documentation, and using Configure, ensures that this is done to the highest standard possible, given the software we have available.

Configure is also very useful to the end user, and this is one of the reason that I've included wether it is supported or not within the compatibility charts that I generate for tested extensions.  By knowing that it is supported the person who will be managing the wiki will know that the extension will be easier to work with and generally makes MediaWiki more accessible - one of the primary goals we have.

I certainly hope to see more extensions become supported by Configure, and a great deal already are, which is fantastic.  Developing support is also a fairly trivial process and does not consume a significant amount of time - for most extensions it will simply be the case of committing an array to the Configure extension settings file.

Avoid conflicts between extensions

When you install a stable version of MediaWiki on a server, spend a lot of time building content and a community, you rely on MediaWiki to function correctly. Every now and again a security update or a new stable version is released and you are encouraged to update your server. To get the best result, many people install MediaWiki extensions. These extensions may continue to work but it may also be that they need to be updated as well.

Updating can be risky business, so you want to make sure in advance that the updated system will work. To do this, you want to test your system with the new release of MediaWiki and all the relevant extensions, or learn from others who are using a similar configuration.

The Extension testing environment provides you with a virtual server where all the components can be tested. At this moment only the 1.13 version of MediaWiki is supported. When an extension is tested, it is important to know if and how it will affect the running other extensions; in a perfect world multiple extensions work together seamlessly, the amount of isolation between extension is something that needs testing.

We have defined four levels of isolation based on the test results. We consider:
Changes needed to the MediaWiki core
Changes to the database structure
The use of JAVA
Changes to Localsettings.php

As you can see on one server multiple MW version can be installed, here only 1.13.1. with a different WIL number.

Kennisnet is using the Extensions testing environment to make sure that its Wikis will continue to run smoothly. This is persistent and ready to use enviroment.

But if you prefer a other version that is possible but then you have to use a virtual server your self. Start the Wikiation_installer and type ls available

The result is :

7:~# wikiation_installer
=== Wikiation installer (v.  15) ===

please type a command and hit enter
help for help
^D, or quit to quit

installer > ls available
Beta-2002-06-25/
Freeze-2002-06-25/
Initial/
Lamprey/
MOSTLY_WORKING/
RANDOM_RELEASETAG/
RC1-2002-07-11/
REL1_10_0/
REL1_10_0RC1/
REL1_10_0RC2/
REL1_10_1/
REL1_10_2/
REL1_10_3/
REL1_10_4/
REL1_11_0/
REL1_11_0RC1/
REL1_11_1/
REL1_11_2/
REL1_12_0/
REL1_12_0RC1/
REL1_12_1/
REL1_12_2/
REL1_12_3/
REL1_13_0/
REL1_13_0RC1/
REL1_13_0RC2/
REL1_13_1/
REL1_13_2/
REL1_13_3/
REL1_1_0/
REL1_2_0/
REL1_2_0rc1/
REL1_2_0rc2/
REL1_2_0rc3/
REL1_2_1/
REL1_2_2/
REL1_2_3/
REL1_2_4/
REL1_2_5/
REL1_2_6/
REL1_3_0/
REL1_3_0beta1/
REL1_3_0beta2/
REL1_3_0beta3/
REL1_3_0beta4/
REL1_3_0beta4a/
REL1_3_0beta5/
REL1_3_0beta6/
REL1_3_1/
REL1_3_10/
REL1_3_11/
REL1_3_12/
REL1_3_13/
REL1_3_14/
REL1_3_15/
REL1_3_16/
REL1_3_17/
REL1_3_18/
REL1_3_2/
REL1_3_3/
REL1_3_4/
REL1_3_5/
REL1_3_6/
REL1_3_7/
REL1_3_8/
REL1_3_9/
REL1_4_0/
REL1_4_1/
REL1_4_10/
REL1_4_11/
REL1_4_12/
REL1_4_13/
REL1_4_14/
REL1_4_15/
REL1_4_2/
REL1_4_3/
REL1_4_4/
REL1_4_5/
REL1_4_6/
REL1_4_7/
REL1_4_8/
REL1_4_9/
REL1_4beta1/
REL1_4beta2/
REL1_4beta3/
REL1_4beta4/
REL1_4beta5/
REL1_4beta6/
REL1_4branch/
REL1_4rc1/
REL1_5_0/
REL1_5_1/
REL1_5_2/
REL1_5_3/
REL1_5_4/
REL1_5_5/
REL1_5_6/
REL1_5_7/
REL1_5_8/
REL1_5_RC1/
REL1_5_RC2/
REL1_5_RC3/
REL1_5_branch/
REL1_5alpha1/
REL1_5alpha2/
REL1_5beta1/
REL1_5beta2/
REL1_5beta3/
REL1_5beta4/
REL1_6_0/
REL1_6_1/
REL1_6_10/
REL1_6_11/
REL1_6_2/
REL1_6_3/
REL1_6_4/
REL1_6_5/
REL1_6_6/
REL1_6_7/
REL1_6_8/
REL1_6_9/
REL1_7_0/
REL1_7_1/
REL1_7_2/
REL1_7_3/
REL1_8_0/
REL1_8_1/
REL1_8_2/
REL1_8_3/
REL1_8_4/
REL1_8_5/
REL1_9_0/
REL1_9_0RC1/
REL1_9_0RC2/
REL1_9_1/
REL1_9_2/
REL1_9_3/
REL1_9_4/
REL1_9_5/
REL1_9_6/
REV1_2/
R_PHASE3REORG/
WIKIMANIA_STEP1/
WITH_CSHARP/
arelease/
initial-udpmcast/
merged_to_schema_work/
merged_to_schema_work2/
merged_to_schema_work3/
merged_to_schema_work4/
merged_to_schema_work5/
merged_to_wikidata_1/
merged_to_wikidata_2/
start/
tug_0_1/
installer >

So enough to choose from.
Summery.
We want to make the extension testing environment as easy as possible to use. So we have the persistent extensions installed ready to use without any effort. But this can never be complete and identical to your situation. So with the virtual server you can install the specific MW version you want to test with. That is a bit more work but very flexible.

So now you are able to test an upgrade to new releases of MediaWiki and also be sure that new extensions will not upset the current smooth operations.

Thank you,

Bernard

Wikiation.nl