MediaWiki Testing

MediaWiki testing under a new name

2009-05-13T19:49:00.000+02:00

With regret I have to announce that the collaboration with Wikiation has come to an end. The root cause are difference of opinion on how an Open Source project is to be run. Wikiation has been very important in the development of the software and this is something we do acknowledge. However there was no effective appreciation for all the work that was done for free and sadly the software did not get enough of a practical application because of directions by Wikiation.

As Wikiation has pulled out of the project and as it has indicated that its name should not be associated with our continued development, the software will now be called "MediaWiki Testing Environment". It is no longer called an "Extension Testing Environment" because it is not only extensions that get tested by the software.

We are now preparing this conversion and, we are preparing the integration of the running of test cases in the MediaWiki Testing Environment.

We expect that we will have more positive news for you in the future.
Thanks,
GerardM

Demoing the Wikiation Extension Testing Environment

2009-04-24T11:41:00.000+02:00

Yesterday I had the pleasure of presenting the extension testing environment to people of the Wikipedia Usability Initiative (UI). It was a pleasure because many of the requirements for the UI can be met by our environment.

What we discussed and/or demoed:

the motives for the Wikiation Extension Testing Environment
testing on different operating systems and (LAMP) configurations
the need for the creation and duplication of test environments
the need for testing environments with different configurations for instance with and without flagged revisions
the installation of extensions and its problems
the ability to script tests and installations in bash or in python
why integrating Selenium makes sense (remote control and grid)
we demoed the installation of an en.wikipedia lookalike and talked about the need for both internal and external testing
the need for sharing the test cases as widely as possible

What I liked in this conversation was the priviledge of having a look into the "UI kitchen", I got the impression that the Wikipedia Usability Initiative is progressing nicely.
Thanks,
GerardM

WikiArticleFeeds

2009-04-12T16:37:00.000+02:00

I am building he WikiEducator test environment and as I mentioned in a previous blog, several of the extensions exist as a text source somewhere on a Wiki. This was also true for WikiArticleFeeds. I am really happy that the developer of WikiArticleFeeds has submitted it to the WMF SVN.

Having WikiArticleFeeds in the WMF SVN, is the first step of Open Source magic. I asked on IRC, the mediawiki-i18n for support and half an hour later an intenationalisation file had been added by IAlex. It is now ready to be supported on translatewiki.net.

I have asked other WikiEducator developers to help me out. The response so far has been great. I love such Open Progress.
Thanks,
GerardM

The WikiEducator environment

2009-04-11T12:37:00.000+02:00

WikiEducator is where the Commonwealth of Learning has an excellent project for free e-learning content; I rate it as best of breed. As the Commonwealth of Nations consists of many countries in all parts of the world, WikiEducator has to take into account the many different cultures in order to provide quality education.

WikiEducator has invested in the development of its Open Source educational environment and this makes it a challenge to build an environment in the Wikiation Extension Testing Environment.

Many of the MediaWiki extensions are not in a code repository system. They are included as a text in a Wiki, sometimes they are just available to be downloaded. This has several drawbacks;

the installer cannot load the code
the code is not internationalised and localised at translatewiki.net
it is not clear if there have been modifications to the code in the past

Even though it is not that hard to create a new extension in the WMF SVN, it is best when this is done either by the developers or by the people behind WikiEducator. When asked typically there is a positive response.

A special case is the LiquidThreads extension. This extension is broken in its latest revision. The testing environment makes it really easy to establish this or to find what effect other extensions have. The COL and Unicef are collaborating, it is fairly trivial to install and test the Uniwiki extensions in a combined environment as well.

The current incarnation of the WikiEducator testing environment can be found here.
Thanks,
GerardM

Jmol

2009-04-09T16:27:00.000+02:00

Jmol is an extension I blogged about in the past. What Jmol does is show what chemicals, proteins look like. It does not only do a good job visualising, it really lightens up pictures that are otherwise rather dull.

Installing Jmol proved to be interesting, it has a prerequisite called StubManager. This is a tool that is essential for a whole range of extensions and it is required to be installed before all the tools that make use of it.

Jmol itself was interesting as well; it installed itself in a directory called MediaWiki and ia directory with java stuff. For the Wikiation Extension Testing Environment, we are looking for a standard way of installing the stuff because that also allows for clean un-installs. Everything is now in the Jmol directory, we now only have to make it work.
Thanks,
GerardM

Environments

2009-04-07T22:02:00.000+02:00

When you test an installer, you need to test the installation of extensions. I made the choice to build copies of MediaWiki environments. My assumption is that the functionality of great environments is what would be of interested to many people.

Referata provides a best practice environment for Semantic MediaWiki. Yaron, the man who runs Refarata is also behind many of the Semantic MediaWiki extensions. Obviously the secret sauce of Referata is Yaron's ability to fix whatever comes his way.

I informed Yaron about the test environment, we found some issues and he fixed them for us.
Thanks,
GerardM

Selenium

2009-04-07T16:54:00.000+02:00

The Wikiation Extension Testing Environment has so far concentrated on server side issues. How do we install MediaWiki, how do we install extensions, how do we run tests serverside. The tests we currently run are parser tests and tests run using the pywikipedia framework.

This is all really nice but the proof of the pudding is in the eating and this is done with browsers. Given our values, tools to do the testing need to be freely licensed as well.

The great thing about Selenium is that it already has many add ons to do serious testing. It allows for automated testing for multiple browsers. It allows for using a grid of systems to do the testing. To me it looks exactly right. I asked MinuteElectron to look at it as well; he is really entheausiastic, he started to convert his work on Uniwiki into Selenium already.

The question is what do you think?
Thanks,
GerardM

Berlin meeting

2009-04-06T08:17:00.001+02:00

The Berlin conference was over before I realised it. I had prepared a presentation, not many people saw it because there was no beamer and, people were more interested in a demo and in discussing what the testing environment is all about.

I demoed the testing environment, I scratch installed the Wikipedia environment many times. I explained the benefits of a shared testing framework, why the ability of installing an environment is so crucial, how we can run the tests the extensions that are installed. Most importantly I explained who the interested parties are.

A developer spends 70 to 90% of his time testing and debugging, quality testing cuts down on that number
Brion needs to do the same tests, so when he has the tests he will be more efficient
Many people run their own MediaWiki installation, they need to know if an extension works for them

For me the most important take home message is what Brion told me: "People want me to look at code, if this code comes with testcases I can run, I am much more likely to have a look". When you consider the number of extensions that are waiting for Brion to look at, extensions that have community support, it is clear that testing and sharing the test-cases is an essential tool in the battle for Brion's attention.
Thanks,
GerardM

Duplicate

2009-03-26T16:17:00.000+01:00

So far we assumed that when you want to test MediaWiki and its extensions for a particular environment, you build a system from scratch, run your tests and assess if the new functionality is an improvement over the old. It turns out that the notion that a particular revision of an extension includes the whole of that extension is not sound. Several extensions can not be installed by specifying their revision.

The ability to test if an extension is complete, can be determined by comparing the difference between the functionality after an upgrade or an installation. To do this you have to be able to save an "old" environment. Another reason why the new duplicate function is useful, is because it allows us to speed up large scale automated testing

Syntax: duplicate MW_installation MW_installation_new

Thanks,
GerardM

WMF_testsystem update

2009-03-25T16:44:00.003+01:00

The WMF_testsystem is one of the MediaWiki instances in the Wikiation Extension Testing Environment. This system is intended to mimick the English Wikipedia.

The way this configuration is build is distinctly different from the English Wikipedia. This system is scratch installed, it does not share the files needed for CentralAuth and MergeAccount, the Cite and the SpecialCite are in separate folders and last but not least, there are a few extensions that I have not figured out how to install yet.

What I really liked was that it took me only five minutes to rebuild my environment once Wikipedia was updated. Three minutes to change my installation script and two minutes to run it.
Thanks,
GerardM

When less is more

2009-03-24T15:23:00.000+01:00

When I enter the command ls revisions.mediawiki: I get a long list of revisions. They are in lifo format and it is a long list. When I enter this commond from within the Wikiation Installer in PuTTY, the list has a length that is too much for the PuTTY buffer. It is possible to limit the output by adding "limit 20". This works fine.

When you run the command from Bash, like this: ./installer.py ls revisions.mediawiki: it is possible to add command separated with a pipe. I tried the less and this worked fine for me. That is until I learned that I could break the pipe by pressing the "q" to quit, that gave a traceback.

The good news is that we can inform you what revisions for MediaWiki are available. It is possible to limit the number of most recent revisions, it is even possible to use bash for this and the pipe will get some more plumming.
Thanks,
GerardM

Proof of the pudding

2009-03-23T08:32:00.003+01:00

One of the objectives of the Wikiation Extension Testing Environment is to have the ability to test the functionality of a specific configuration. A prerequisite is the ability to built an environment that has the same configuration.One of the missing components was the ability to install a specific revision of MediaWiki itself. The screen shot demonstrates our new ability to do this.

The way we install our extensions is different from doing it manually. We expect to be able to install and uninstall extensions and consequently each extension needs to be in its own directory. For the Cite and the Special:Cite extension it meant that they are split in seperate directories. They install successfully and do not break MediaWiki so we assume that this is fine.

At this moment we assume that things are fine when the system installs and when we can save a page. Running tests will be our goal once we can reliably build our testing environment.
Thanks,
GerardM

Installng Metavid

2009-03-20T21:55:00.000+01:00

As we are developing and testing the Wikiation Installer, we are recreating relevant configurations. Metavid is the open video archive of the US Congress and the Metavid software will be the basis of video support for MediaWiki. People are interested in learning about these software developments and being able to build such environment is fun.

The metavid profile brought several things to me that were new. There are programs external to MediaWiki, programs that are not part of the LAMP stack that are prerequistites. We do not want to install such programs from within the installer but we do want to be able to test for such prerequistites. At this stage, we do not test for this, we leave this on our "wish list".

I had to install a SQL script in order to install metavid tables in the MediaWiki database. This was beyond me. Kim helped me out and now I have an example that I can copy. Now that the files install, I also needed a way to know that the files had installed. Michael Dale, the developer of Metavid suggested to use phpmyadmin. This was an attractive suggestion but, it does add to the clutter in what should be a clean environment.

What to do.. With hindsight the answer was obvious; one of the tests checks if new tables have been added to the database. So eating our own dogfood was in order. The isolation check was revived and indeed tables proved to have been added for Metavid.

I may have done a good job. I do not know as I do not how to test the functionality of this metavid environment.

Thanks,

GerardM

Revisions and Tags

2009-03-19T13:26:00.000+01:00

When you are installing MediaWiki or MediaWiki extensions, it is often crucial to be able to install a particular revision or for a particular "tag". I am quite pleased to announce that the Wikiation installer is now able to install both for specific revisions or tags.

This functionality is currently very much bleeding edge. I do not know yet how to apply the tags to the scripts that I already wrote. So I will do some more documentation and some other work...
Thanks,
GerardM

How to install RSS_Reader ?

2009-03-19T00:02:00.000+01:00

RSS_Reader is an extension. It must be because it says so on the MetavidWiki and on the MediaWiki wiki. The problem that I face is that it has not been packaged as an extension.

That is a shame for several reasons.

I cannot install it
messages do not get internationalised and localised at translatewiki.net
it prevents me from building a complete MetavidWiki test wiki with the installer

The information on the MetavidWiki says that it is "version 0.2.3". Now what if I need to run the software on a previous release. I may need "version 0.2.2" and I cannot get it because it is not in SVN.

I confess that I am not a developer. But I am quite sure that when you follow the instructions, you have something that might actually work. A developer, a real one, someone who Brion trust to submit to SVN, could then make it an extension so that I could do my "monkey see, monkey do" routine.
Thanks,
GerardM

Installing the UNIWIKI extensions

2009-03-17T21:34:00.001+01:00

When you want to test software, you need to be able to set up an environment quickly and repeatedly. The Uniwiki extensions are tested by customers of Wikiation, a programmer is working on the code so it makes sense to be able to create this environment quickly.

Programming in bash is something that I did a really long time ago, so it was hard work for me to get the install scripts right. Once I had the first one working, the other nine were easy.

I am really fond of the Configure extension so I always install that one. The only thing that I have done is install the software and I leave it to you to test it.
Thanks,
GerardM

Documentation ....

2009-03-16T19:18:00.001+01:00

Documentation is almost universally something that you may do at the end. As such it is quite similar to testing. Python has this nice little tool called pydoc. It helps you generate documentation as you are working the code and as a consequence it provides a great incentive to work on documentation from the start.

One of the nice features is a web front end to the documentation. If you are interested, have a look at our documentation. This is the documentation of the bleeding edge, so you may see things that have not even been committed to SVN. To get an updated to the latest version in SVN, use update_self from within the installer.
Thanks,
GerardM

Installing a MediaWiki system

2009-03-16T09:03:00.001+01:00

How long does it take to install a new MediaWiki system with Semantic MediaWiki? It takes me a minute and yes, I cheated because I automated the process. This kind of automation is extremely important to us because it is the basis for running automated tests.

Thanks,

GerardM

Installing SemanticForms

2009-03-15T11:18:00.000+01:00

I created an installer script for SemanticForms, everything should work but it did not. I could not find what the problem was so I asked MinuteElectron to help me out. After some research he found that it had to do with the order in which the SemanticMediaWiki and the SemanticForms were loaded; they are loaded in alphabetical order so the SemanticForms came in first.

I have created a hack to get around this; I have renamed the SemanticForms.settings.php to xSemanticForms.settings.php. The documentation is clear; Semantic MediaWiki has to be installed and in this way MediaWiki is aware of this. Adding an x does the trick.

I wonder if this is one of those temporary solutions that will prove to be permanent.
Thanks,
GerardM

Installing SemanticResultFormats

2009-03-15T00:00:00.002+01:00

Programming is what I used to do a lot. The reason why I am programming again is because I am working on the documentation for the Wikiation Installer. I cannot document properly what I do not really understand and programming some installer scripts is one way of getting to grips with the issues.

I am testing in the "Fosdem" environment, this is where developers may test the existing functionality... I did install Semantic MediaWiki and I decided that I wanted to install SemanticResultFormats as well. I read the documentation and I found that due to a non standard include, I had to write an installer script. This is annoying but trivial

I tested it, the version special page reports it as being installed so I was good. However, it struck me that SemanticResultFormats has in SemanticMediaWiki a prerequisite. So I dabbled some more in the download.sh only to come to the conclusion that this is not the place where prerequisites are to be tested. The reason for this is that the include would be excercised anyway. When SemanticMediaWiki is not there, the whole installation should abort.

Writing installation scripts does help me understand the environment, it will help me write better documentation and in the mean time, I find where the software can use some more refinenment. I do confess that I am looking for the corner cases and so far I am really pleased how the installer is improving.
Thanks,
GerardM

Working from a template

2009-03-14T15:55:00.000+01:00

So I was bold and decided to write the install and uninstall script for another extension. I chose ConfirmAccount because of its similarity to LiquidThreads.

There were many lessons for me to learn:

how do I duplicate a directory - mkdir and cp -f
where can I find the information about the MySQL files that are created
how do I create a tar file with the directory in it - tar -cvvf confirmaccount.tar ConfirmAccount.install/
how do I get this tar file from the system so that I can send it by mail
what did I forget ?

It is not that hard but it really helps when things are done in a standard way.
Thanks,
GerardM

Installing LiquidThreads

2009-03-14T12:17:00.003+01:00

When you install LiquidThreads with the naive installer, it installs fine and it does not break MediaWiki. When you go to a talk page the system still breaks because the "thread" table is missing. When you read the documentation, it is quite obvious that LiquidThreads would fail in this way. There is a script that helps you to install the tables and as I expected that it would be relatvely easy to create a script for the complete install and uninstall of LiquidThreads I looked into this.

I asked MinuteElectron to help me out and provide me with an example, he added one line to the standard "download.sh". For the uninstall he added a few lines to the "uninstall.sh" and added a file with the names of the tables that were to be dropped.

I now have a template for creating install scripts for extensions. I will have to ask Kim or MinuteElectron to update SVN with my scripts because if I were Brion, I would not trust me with SVN either.
Thanks,
GerardM

Working with the Installer

2009-03-13T14:02:00.002+01:00

I wanted to test a function of the installer. So I went to the fosdem.wikiation.nl and ran the "update_self" functionality. This broke the installer. So I deleted the installer directory and installed the installer again from the documentation.

This worked fine. To my delight I got the following welcome message:

****************************************
=== Wikiation installer (v. 48376) ===

(last known safe version: 48307)
Interactive mode. Automated testing is disabled.

please type a command and hit enter
help for help
^D, or quit to quit
****************************************

So there is now the latest version or the latest safe version. Now I have to learn how to install the revision that I want to test because that needs to end up in the documentation.
Thanks,
GerardM

Debugging Extension Installation Scripts

2009-03-12T11:00:00.005+01:00

When creating extension installation scripts for the extension testing environment, it can be difficult to analyse where issues are in the script. For example, when we were creating the Semantic MediaWiki installation script we did not know why the setup program was not running correctly. Previously, the installer would not output anything from install files and so no debugging was possible. Now, there is a debug flag which allows us to see detailed information - enabling us to find and fix issues far more efficiently than before.

To use this flag, add "debug=True" to your settings.py file.

Getting ready for the next level of functionality

2009-03-08T12:07:00.001+01:00

When the Wikiation Extension Testing Environment currently installs an extension, it uses the latest version of that extension. This is fine when you are testing the very latest version of MediaWiki.

We want to provide better support for all the wikis who are using stable versions of MediaWiki. To do this, we have to be able to install a particular version of an extension, the latest version of an extension for a particular release or for a stable version of an extension for a particular release.

The other thing we are actively working on is increasing the number of extensions that can be installed by the installer. MinuteElectron asked for the output of installer scripts. As Kim was already collecting this information, it was easy to add a debug flag that makes this information available. This will make it easier to get extensions to install properly.

Another improvement is that there is now a default settings handler, this is intended to prevent future updates of the software using the "update_self" functionality from breaking. This makes the installer more robust.

Our intent is that the installer may be used for installing both production and test environments of MediaWiki. As the software creates and destroys instances of a MediaWiki wiki, it is vital that you do not experiment on a production system. It is equally important that you are aware that at this stage the software is not ready for production systems.
Thanks,
GerardM

258 out of 359 extensions can plausibly be installed with the naive installer

2009-02-26T09:24:00.001+01:00

We are developing a test environment for MediaWiki. In order to test MediaWiki and its extensions we have to be able to install them. The "naive extension installer" makes a good faith approach to installing an extension. At this stage we call an installation a success when the install does not break MediaWiki.

NB We have installed the latest version of an extension on the latest version of MediaWiki at the time of testing.

When 72% of the extension can be plausibly installed, a lesser percentage will still need all kinds of things done before it is functional. All kind of other things may need to be addressed like configurations and what not. We already support the Configure extension for that ...

In all this info glut about installing extensions there is one other really important point: this is the first automated test of all extensions. This is the very start were we start to inform you with test results.
Thanks,
GerardM

PS Is _your_ extension part of the 258 installable ones ???

New main page

2009-02-24T14:09:00.001+01:00

We have updated the [[Main page]] of the Wikiation Extension Testing Environment Wiki. It is now more of a portal to the information that we provide on our wiki.

Please let us know what needs to be clarified. Please let us know what is missing. Obviously it is very much a work in progress, when we get a lot input from you, we do not need to repeat the message that we want you to be involved that often. :)
Thanks,
GerardM

Installing with Configure present - Edittools fun

2009-02-24T10:09:00.003+01:00

I am really pleased announcing Configure support for the installation of extensions. Now when you install an extension, it will leave the enabling and the configuration to Configure. I asked for it, it is important and now I have to come to grips with it.

One of the things that it has is a button to enable an extension.. That is good. It does not necessarily help me understand how things work. The Edittools is one of my favourites and it needs CharInsert. So I enabled it and it does not give me my drop down box that should be there..

One of the friendly people at IRC told me about the existence of Edittools.js but is was not clear if this was what I had to install.

So it seems that there is functionality that needs all kinds of components, there is the message, there is the Edittools message, There is the Edittools.js and there is the CharInsert extension.

So how do I activate the Edittools.js and what to do with: "the dropdown is added by Edittools.js if document.getElementById('specialchars'). This is a bit inefficient though, and it is rather outdated code". I know for a fact that it is used on LOADS of MediaWiki servers..
Thanks,
GerardM

More extensions installed

2009-02-23T22:28:00.000+01:00

We have made some great progress with the testing software.. Yesterday, the Babel extension failed to work properly. Some research showed that the current virtual machine runs an older version of PHP. Now that it works, we learned that the Babel extension needs some changes to the CSS.

It may sound odd, but working with the code, testing and learning what more needs to be done works well for us. We have made a lot of progress.

Over the weekend MinuteElectron and Denny have worked on an install script for Semantic MediaWiki. This is basic Semantic MediaWiki only. It installs clean and it uninstalls clean.

The next thing to get on-line is the Configure extension; we hope that this will give us even more possibilities in our testing environment.
Thanks,
GerardM

First iteration of a naive extension installer

2009-02-22T12:32:00.001+01:00

In the Wikiation Extension Testing Environment, we want to test extensions. These extensions have to be installed. To install an extension, you have to have a script to install a script ... Hmmm, that sound like work. Who is going to do that... Hmmm, we could write a "naive installer", that would solve 60/80% of the required scripts.

So Kim wrote a naive installer and it just does do a naive install. In the fosdem.wikiation.nl environment I abused the Brion wiki and installed the InputBox and the Babel extension. One of the two worked, both should have worked. Testing is nice.
Thanks,
GerardM

The Wikiation extension testing environment is getting ready for business

2009-02-19T01:06:00.000+01:00

The Wikiation extension testing environment has been made available in the WMF SVN.This is good news because more people can see what we are actually doing and what we already achieved.

At FOSDEM we demonstrated the environment to Brion. We promised him that an environment would be made available for testing. The environment has now been made available to him at http://fosdem.wikiation.nl. It is a virtual server where MediaWiki and extensions to MediaWiki can be installed and tested.

The objective of the environment is to have a way to determine if an extension works well on a specific platform. and in combination with a specific MediaWiki release. It is not been clear at all what extension works well with what stable release of MediaWiki.

In order to be able to test, we have to be able to install MediaWiki and its extensions. We are getting good at installing MediaWiki. We are now working on getting better at installing extensions. We need a generic script for the installation of extensions, then we need to be able to install a particular version from a particular branch and then we also want to be able to install a "stable" version.

Our idea of a stable version is that it does not crash MediaWiki and that it passes the tests we have for the extension and the system. This means that when an extension does nothing, it is acceptable ... Now as I understand it, it take an agile mind to appreciate why this is a good idea.

When I have time, when I am not blogging, or doing admin, or sleeping, I am also working on documentation. I documented how to install the Wikiation Installer ... I used bash for that. :) who needs documentation, the code makes it obvious :)

Thanks,
GerardM

Why perform functional tests

2009-02-15T08:41:00.000+01:00

When you are in a test environment, you can expect that things are not completely functional. It does not mean that the environment is completely broken.

pecific behaviour can be expected from an extension. From Liquid Threads it can be expected that you can reply to a thread. It is possible to create a test for this.

The benefits are obvious; no broken code goes live in the test environment, the test environment does not get corrupted and the people who test are not bothered by functionality that is broken. Essentially some time needs to be invested to write a test up front. The return on investment is huge because it is not only the developers time that is saved, no need to restore the test environment, but also the time of the people who test.

The time of the people who test is typically not valued. It is however a sign of respect to the testers that obvious tests are performed first before they are exposed to new functionality.
Thanks,
GerardM

Fosdem 2009

2009-02-04T22:46:00.041+01:00

Brion will be one of the speakers at FOSDEM. It is great to meet Brion at FOSDEM. What I find really exciting is that we will demonstrate our ExtensionTesting environment to him. Let me know what we are currently testing:

Installation routines for MediaWiki
Installation routines for MediaWiki extensions
Setting up test cases for testing HTML output for MediaWiki or extensions
Running automated test cases
Uninstall routines for MediaWiki
Uninstall routines for MediaWiki extensions

When you get the impression that most of this can be run in a batch from a script, that is exactly our intention. When you get the impression that an installation routine can be used in a production environment, that should be possible as well.

We are working hard to have a live demonstration for Brion. My main worry at the moment is the availability of Internet.
Thanks,
GerardM

Automated Extension Testing

2009-01-18T16:49:00.002+01:00

So far all the extension testing that I have been doing were manual tests; I defined a procedure, followed it, and noted down the discrepancies between the expected and the actual behaviour. For certain extensions this is the only feasible way, but when a given input creates a given output this whole process can be automated.

Wikiation is developing a testing environment that works in a similar way to the MediaWiki core parser test system but it is more flexible. It seemed a good moment to test the Cite extension. As I am now able to use automated tests, I can repeat this more often and be much more efficient.

Improved reliability and accuracy is why you want to use automated tests. It prevents human errors and doing the same boring stuff is something computers are good at. Making sure that new releases of MediaWiki work with the installed extensions is a lot easier. All the tests can be run again and again, so when things continue to work like they used to, there is a good indication that everything is fine.

The Configure Extension and Its Relevance

2009-01-10T14:35:00.000+01:00

Recently I have started testing the Configure extension, it is very valuable to our project for a variety of reasons - and indeed quite unlike most of the other extensions we test.

Primarily it will help to ensure that testing is more comprehensive, if extensions are supported by configure then it reduces the chance that a software tester (such as myself) will miss out testing certain configuration settings. Making sure that all configuration settings are tested at least independently, and even better together (of course, this isn't always feasible), is integral to the validity of results. Working by both reading the settings in the extension's documentation, and using Configure, ensures that this is done to the highest standard possible, given the software we have available.

Configure is also very useful to the end user, and this is one of the reason that I've included wether it is supported or not within the compatibility charts that I generate for tested extensions. By knowing that it is supported the person who will be managing the wiki will know that the extension will be easier to work with and generally makes MediaWiki more accessible - one of the primary goals we have.

I certainly hope to see more extensions become supported by Configure, and a great deal already are, which is fantastic. Developing support is also a fairly trivial process and does not consume a significant amount of time - for most extensions it will simply be the case of committing an array to the Configure extension settings file.

Avoid conflicts between extensions

2009-01-09T21:20:00.000+01:00

When you install a stable version of MediaWiki on a server, spend a lot of time building content and a community, you rely on MediaWiki to function correctly. Every now and again a security update or a new stable version is released and you are encouraged to update your server. To get the best result, many people install MediaWiki extensions. These extensions may continue to work but it may also be that they need to be updated as well.

Updating can be risky business, so you want to make sure in advance that the updated system will work. To do this, you want to test your system with the new release of MediaWiki and all the relevant extensions, or learn from others who are using a similar configuration.

The Extension testing environment provides you with a virtual server where all the components can be tested. At this moment only the 1.13 version of MediaWiki is supported. When an extension is tested, it is important to know if and how it will affect the running other extensions; in a perfect world multiple extensions work together seamlessly, the amount of isolation between extension is something that needs testing.

We have defined four levels of isolation based on the test results. We consider:
Changes needed to the MediaWiki core
Changes to the database structure
The use of JAVA
Changes to Localsettings.php

As you can see on one server multiple MW version can be installed, here only 1.13.1. with a different WIL number.

Kennisnet is using the Extensions testing environment to make sure that its Wikis will continue to run smoothly. This is persistent and ready to use enviroment.

But if you prefer a other version that is possible but then you have to use a virtual server your self. Start the Wikiation_installer and type ls available

The result is :

7:~# wikiation_installer
=== Wikiation installer (v. 15) ===

please type a command and hit enter
help for help
^D, or quit to quit

installer > ls available
Beta-2002-06-25/
Freeze-2002-06-25/
Initial/
Lamprey/
MOSTLY_WORKING/
RANDOM_RELEASETAG/
RC1-2002-07-11/
REL1_10_0/
REL1_10_0RC1/
REL1_10_0RC2/
REL1_10_1/
REL1_10_2/
REL1_10_3/
REL1_10_4/
REL1_11_0/
REL1_11_0RC1/
REL1_11_1/
REL1_11_2/
REL1_12_0/
REL1_12_0RC1/
REL1_12_1/
REL1_12_2/
REL1_12_3/
REL1_13_0/
REL1_13_0RC1/
REL1_13_0RC2/
REL1_13_1/
REL1_13_2/
REL1_13_3/
REL1_1_0/
REL1_2_0/
REL1_2_0rc1/
REL1_2_0rc2/
REL1_2_0rc3/
REL1_2_1/
REL1_2_2/
REL1_2_3/
REL1_2_4/
REL1_2_5/
REL1_2_6/
REL1_3_0/
REL1_3_0beta1/
REL1_3_0beta2/
REL1_3_0beta3/
REL1_3_0beta4/
REL1_3_0beta4a/
REL1_3_0beta5/
REL1_3_0beta6/
REL1_3_1/
REL1_3_10/
REL1_3_11/
REL1_3_12/
REL1_3_13/
REL1_3_14/
REL1_3_15/
REL1_3_16/
REL1_3_17/
REL1_3_18/
REL1_3_2/
REL1_3_3/
REL1_3_4/
REL1_3_5/
REL1_3_6/
REL1_3_7/
REL1_3_8/
REL1_3_9/
REL1_4_0/
REL1_4_1/
REL1_4_10/
REL1_4_11/
REL1_4_12/
REL1_4_13/
REL1_4_14/
REL1_4_15/
REL1_4_2/
REL1_4_3/
REL1_4_4/
REL1_4_5/
REL1_4_6/
REL1_4_7/
REL1_4_8/
REL1_4_9/
REL1_4beta1/
REL1_4beta2/
REL1_4beta3/
REL1_4beta4/
REL1_4beta5/
REL1_4beta6/
REL1_4branch/
REL1_4rc1/
REL1_5_0/
REL1_5_1/
REL1_5_2/
REL1_5_3/
REL1_5_4/
REL1_5_5/
REL1_5_6/
REL1_5_7/
REL1_5_8/
REL1_5_RC1/
REL1_5_RC2/
REL1_5_RC3/
REL1_5_branch/
REL1_5alpha1/
REL1_5alpha2/
REL1_5beta1/
REL1_5beta2/
REL1_5beta3/
REL1_5beta4/
REL1_6_0/
REL1_6_1/
REL1_6_10/
REL1_6_11/
REL1_6_2/
REL1_6_3/
REL1_6_4/
REL1_6_5/
REL1_6_6/
REL1_6_7/
REL1_6_8/
REL1_6_9/
REL1_7_0/
REL1_7_1/
REL1_7_2/
REL1_7_3/
REL1_8_0/
REL1_8_1/
REL1_8_2/
REL1_8_3/
REL1_8_4/
REL1_8_5/
REL1_9_0/
REL1_9_0RC1/
REL1_9_0RC2/
REL1_9_1/
REL1_9_2/
REL1_9_3/
REL1_9_4/
REL1_9_5/
REL1_9_6/
REV1_2/
R_PHASE3REORG/
WIKIMANIA_STEP1/
WITH_CSHARP/
arelease/
initial-udpmcast/
merged_to_schema_work/
merged_to_schema_work2/
merged_to_schema_work3/
merged_to_schema_work4/
merged_to_schema_work5/
merged_to_wikidata_1/
merged_to_wikidata_2/
start/
tug_0_1/
installer >

So enough to choose from.
Summery.
We want to make the extension testing environment as easy as possible to use. So we have the persistent extensions installed ready to use without any effort. But this can never be complete and identical to your situation. So with the virtual server you can install the specific MW version you want to test with. That is a bit more work but very flexible.

So now you are able to test an upgrade to new releases of MediaWiki and also be sure that new extensions will not upset the current smooth operations.

Thank you,

Bernard

Wikiation.nl

You can bank on it, together you do not need to invest as much

2008-12-17T16:26:00.000+01:00

The Stanton project for MediaWiki usability will start in a few weeks. This does not mean that nothing is happening. Today I learned that a bank has spend considerable effort in introducing the UNICEF extensions. One of their programmers has spend considerable effort to improve the software. I have been told that much of this has been fed back to UNICEF. As the programmer is leaving the bank, I was happy to learn that he is allowed to take with him all the code involved.

This programmer does so far not have SVN commit rights. I learned that he is going to work for another company.. Another company interested in implementing the UNICEF extensions.

What this proves to me, is the need for collaboration between all these organisations that develop MediaWiki. It is more effective to work together. Better results can be achieved for the same costs when coordination becomes a priority..

I had a word with Erik.. He told me that MediaWiki already has something like the CreatePage extension in core. It is part of a search result...

What else happens when you click on Uniwiki or what else happens when you click on Create Page..
Thanks,
GerardM

Uniwiki extensions

2008-12-02T18:37:00.000+01:00

This week I finished testing the Uniwiki extensions. Some of the ideas that they promote are quite interesting and definitely something to keep an eye on in the future.

I run a wiki with my Dad and many of the users are quite unfamiliar with the wikitext structure. The Uniwiki Generic Edit Page extension, for example, would definitely help to make the MediaWiki environment friendlier towards them. Even if it is only a small part of the page, removing heading and category tags causes the environment to be so much easier to use; as an experienced editor even I would be likely to opt for using it on my wikis.

There are a few bugs that need working on in a couple of the Uniwiki extensions, but seeing some of them rolled out to Wikimedia wikis would be even nicer and help those 80% of smaller wikis that don’t have as much contribution as we’d like.

Testing against "head"

2008-11-27T10:03:00.001+01:00

The extension testing environment is first and foremost intended to test the functionality of extensions that are to be run on wikis that use the stable versions of the MediaWiki software. There are however extensions like the CreatePage extensions, that have been developed outside the Wikimedia Foundation and that would provide a welcome addition to the WMF projects.

In order to provide the right arguments for such an adoption, it makes sense to test extensions on "head" as well. This argument I posed to Bernard, and he has graciously granted my request. Kim has been asked to make it so.

Several people use the testing environment for their purposes. I learned that even Translate is being tested. If you have a need for testing your extension, the environment can be made available to you too.
Thanks,
GerardM

What stable releases to support?

2008-11-22T12:55:00.000+01:00

I was asked what releases do we support in our MediaWiki extension testing. The question is not so much against what releases we can test against, the question is, what gives the best return of investment.

At this moment, the Wikimedia Foundation supports the 1.12 and 1.13 stable releases. People are urged to be as up to date as possible. To stay up to date, people need to know if the extension they have installed will work in these releases.This is the information that we aim to provide.

By concentrating on the stable versions, there is a combination of technical and localisation support for the core MediaWiki product. For extensions there is plenty of room for improvement; the localisation is currently not supported for previous releases and it is less clear what version of the extension works on any given release.

When you want to learn if an extension changed in its behaviour compared to an old version of MediaWiki, you too can test extensions in our environment. When you want to learn if your extension will work against "head", the test environment is there for you as well.
Thanks,
GerardM

Betawiki supports Uniwiki extensions

2008-11-19T16:25:00.001+01:00

The Uniwiki extensions were developed by UNICEF. They were developed based on usability studies. Studies that found that that 100% of newbies were not able to create a new article in a MediaWiki environment, studies that help explain why so many of our projects do not grow as we want them to do.

We have been doing some testing, and we have found that this software does not work against the bleeding edge software used by the Wikimedia Foundation. Siebrand had a look and fixed the localisation, MinuteElection has a look and fixed a bug.

We are currently talking with Kennisnet and we aim to get in contact with UNICEF. We really think it is important to get more usability into MediaWiki this will help the smallest 80% of our projects a lot.

In the meantime, I am really happy to announce that Betawiki now supports the localisation or the 12 Uniwiki extensions. In anticipation of finding support for these extensions, Siebrand has opened up the localisation for these important extensions.
Thanks,
GerardM

Getting started with testing

2008-11-19T14:00:00.000+01:00

Wikiation has developed the ExtensionTesting software and environment. They have put a lot of effort in making sure that MediaWiki extensions can be tested in a structured way, in a way that helps all users of a MediaWiki installation.

When people create an environment like this, it is important that it provides great, inviting tools that make it almost compulsory to use. To help Wikiation achieve this goal, Stichting Open Progress has sought MinuteElectron to help test drive this environment and test an initial set of extensions.

There are not that many extensions that we have a special interest in. You can suggest extensions for us to test. Particularly requests that include information of what needs to be tested will be of our interest.
Thanks,
Gerard Meijssen
Stichting Open Progress

What tools do you need?

2008-11-17T23:41:00.000+01:00

Currently we have:

virtual servers as test environments (Instead of making things bulletproof, we make things easy to re-spawn)
simple straightforward diff-tests to flag potential situations where extension installations stomping on each other.
server side regression testing.

What else do we need?

Extension testing

2008-11-17T18:04:00.000+01:00

MediaWiki provides a rich environment to its many users. Because of its open source nature, many people develop functionality that is not provided out of the box.

Wikiation does wiki hosting for organisations. When creating a new extension changes has to be made. Quality within organisation is of very high interest. So if a extension has been modified in one detail, the complete extension has to be tested again. Because this change may create an other bug. That is not acceptable. So testing must be intensive. So when there are changes, testing is required again.

Brion Vibber, the CTO of the Wikimedia Foundation, releases several times a year new stable versions of the software. Its recommended to upgrade installations because improved functionality, security and localisations become available. This does results, for each new MediaWiki version, changes to existing extension which may not work any more as they did in the previous version.
So if there are external changes extension testing is required.

There are more then 1000 MediaWiki extensions If a customer of Wikiation does have an interest in a specific extension, Wikiation would like to make it available. But does the exension work as expected? Does the extension make modifications in the core of MediaWiki which might influence the behaviour of the rest of the system? So when a extension written by others is to be used, extension testing is required.

All tree cases are about quality. Does it work as it supposed to work? Is it reproducible. Will my current wikipage, with a specific extension, produce a identical result with a new version of the extension. People are very good in creative tasks, but quite bad in working accurately. Proper testing requires accurate work. And if you want really quality each change requires that everything is tested again.

This task is so huge that a human hardly can do it. The most often used solution is, less testing. That does work but it has to be paid by less quality. Wikiation did search if automated MediaWiki extension testing did exists. It did not. Because of the quality requirements, Wikiation decided to develop its own extension testing system. Each extension on any MW version has to be tested in an automated way. With predefined test sets for specific extensions in specific situation a predefined result has to be generated by the extension. If it does, the status is Ok otherwise it fails. A failure may be an error, but it also may be the wished for improvement. Only a human can be the judge of this.

Wikiation does use MediaWiki software as a hosting company, it could keep the results of this extension testing all to it self. But I do feel it is good to return something to the MediaWiki community. So this extension testing is free to use. The developed software is provided as open source under the GNU 3 license. The documentation and test results at http://extensiontesting.wikiation.nl is based on the Creative Commons Licence. And if you need a CleanMachine for testing, for a very short period, you can use it free of charge. The only thing within extention testing were all rights are reserved is the extension testing logo. I hope you can live with that.

I am not really a blogger. I did start the extension testing. GerardM took the initiative to start this blog. I do thank GerardM also for his help to localise the extensions Wikiation has build.

There is still much to do. Please do your testing, and use the test results others have made. If quality is important to you, join.