MediaWiki testing under a new name

With regret I have to announce that the collaboration with Wikiation has come to an end. The root cause are difference of opinion on how an Open Source project is to be run. Wikiation has been very important in the development of the software and this is something we do acknowledge. However there was no effective appreciation for all the work that was done for free and sadly the software did not get enough of a practical application because of directions by Wikiation.

As Wikiation has pulled out of the project and as it has indicated that its name should not be associated with our continued development, the software will now be called "MediaWiki Testing Environment". It is no longer called an "Extension Testing Environment" because it is not only extensions that get tested by the software.

We are now preparing this conversion and, we are preparing the integration of the running of test cases in the MediaWiki Testing Environment.

We expect that we will have more positive news for you in the future.
Thanks,
       GerardM

Demoing the Wikiation Extension Testing Environment

Yesterday I had the pleasure of presenting the extension testing environment to people of the Wikipedia Usability Initiative (UI). It was a pleasure because many of the requirements for the UI can be met by our environment.


What we discussed and/or demoed:

• the motives for the Wikiation Extension Testing Environment
  
• testing on different operating systems and (LAMP) configurations
  
• the need for the creation and duplication of test environments
• the need for testing environments with different configurations for instance with and without flagged revisions
  
• the installation of extensions and its problems
  
• the ability to script tests and installations in bash or in python
• why integrating Selenium makes sense (remote control and grid)
  
• we demoed the installation of an en.wikipedia lookalike and talked about the need for both internal and external testing
• the need for sharing the test cases as widely as possible
  

What I liked in this conversation was the priviledge of having a look into the "UI kitchen", I got the impression that the Wikipedia Usability Initiative is progressing nicely.
Thanks,
       GerardM

WikiArticleFeeds

I am building he WikiEducator test environment and as I mentioned in a previous blog, several of the extensions exist as a text source somewhere on a Wiki. This was also true for WikiArticleFeeds. I am really happy that the developer of WikiArticleFeeds has submitted it to the WMF SVN.

Having WikiArticleFeeds in the WMF SVN, is the first step of Open Source magic. I asked on IRC, the mediawiki-i18n for support and half an hour later an intenationalisation file had been added by IAlex. It is now ready to be supported on translatewiki.net.

I have asked other WikiEducator developers to help me out. The response so far has been great. I love such Open Progress.
Thanks,
      GerardM

The WikiEducator environment

WikiEducator is where the Commonwealth of Learning has an excellent project for free e-learning content; I rate it as best of breed. As the Commonwealth of Nations consists of many countries in all parts of the world, WikiEducator has to take into account the many different cultures in order to provide quality education.

WikiEducator has invested in the development of its Open Source educational environment and this makes it a challenge to build an environment in the Wikiation Extension Testing Environment.

Many of the MediaWiki extensions are not in a code repository system. They are included as a text in a Wiki, sometimes they are just available to be downloaded. This has several drawbacks;

• the installer cannot load the code
• the code is not internationalised and localised at translatewiki.net
• it is not clear if there have been modifications to the code in the past

Even though it is not that hard to create a new extension in the WMF SVN, it is best when this is done either by the developers or by the people behind WikiEducator. When asked typically there is a positive response.

A special case is the LiquidThreads extension. This extension is broken in its latest revision. The testing environment makes it really easy to establish this or to find what effect other extensions have. The COL and Unicef are collaborating, it is fairly trivial to install and test the Uniwiki extensions in a combined environment as well.

The current incarnation of the WikiEducator testing environment can be found here.
Thanks,
     GerardM

Jmol

Jmol is an extension I blogged about in the past. What Jmol does is show what chemicals, proteins look like. It does not only do a good job visualising, it really lightens up pictures that are otherwise rather dull.

Installing Jmol proved to be interesting, it has a prerequisite called StubManager. This is a tool that is essential for a whole range of extensions and it is required to be installed before all the tools that make use of it.

Jmol itself was interesting as well; it installed itself in a directory called MediaWiki and ia directory with java stuff. For the Wikiation Extension Testing Environment, we are looking for a standard way of installing the stuff because that also allows for clean un-installs. Everything is now in the Jmol directory, we now only have to make it work.
Thanks,
       GerardM

Environments

When you test an installer, you need to test the installation of extensions. I made the choice to build copies of MediaWiki environments. My assumption is that the functionality of great environments is what would be of interested to many people.

Referata provides a best practice environment for Semantic MediaWiki. Yaron, the man who runs Refarata is also behind many of the Semantic MediaWiki extensions. Obviously the secret sauce of Referata is Yaron's ability to fix whatever comes his way.

I informed Yaron about the test environment, we found some issues and he fixed them for us.
Thanks,
       GerardM

Selenium

The Wikiation Extension Testing Environment has so far concentrated on  server side issues. How do we install MediaWiki, how do we install extensions, how do we run tests serverside. The tests we currently run are parser tests and tests run using the pywikipedia framework.

This is all really nice but the proof of the pudding is in the eating and this is done with browsers. Given our values, tools to do the testing need to be freely licensed as well.

The great thing about Selenium is that it already has many add ons to do serious testing. It allows for automated testing for multiple browsers. It allows for using a grid of systems to do the testing. To me it looks exactly right. I asked MinuteElectron to look at it as well; he is really entheausiastic, he started to convert his work on Uniwiki into Selenium already.

The question is what do you think?
Thanks,
       GerardM

Berlin meeting

The Berlin conference was over before I realised it. I had prepared a presentation, not many people saw it because there was no beamer and, people were more interested in a demo and in discussing what the testing environment is all about.

I demoed the testing environment, I scratch installed the Wikipedia environment many times. I explained the benefits of a shared testing framework, why the ability of installing an environment is so crucial, how we can run the tests the extensions that are installed. Most importantly I explained who the interested parties are.

• A developer spends 70 to 90% of his time testing and debugging, quality testing cuts down on that number
• Brion needs to do the same tests, so when he has the tests he will be more efficient
• Many people run their own MediaWiki installation, they need to know if an extension works for them

For me the most important take home message is what Brion told me: "People want me to look at code, if this code comes with testcases I can run, I am much more likely to have a look".  When you consider the number of extensions that are waiting for Brion to look at, extensions that have community support, it is clear that testing and sharing the test-cases is an essential tool in the battle for Brion's attention.
Thanks,
        GerardM

Duplicate

So far we assumed that when you want to test MediaWiki and its extensions for a particular environment, you build a system from scratch, run your tests and assess if the new functionality is an improvement over the old. It turns out that the notion that a particular revision of an extension includes the whole of that extension is not sound. Several extensions can not be installed by specifying their revision.

The ability to test if an extension is complete, can be determined by comparing the difference between the functionality after an upgrade or an installation. To do this you have to be able to save an "old" environment. Another reason why the new duplicate function is useful, is because it allows us to speed up large scale automated testing

Syntax:  duplicate MW_installation MW_installation_new

Thanks,
      GerardM

WMF_testsystem update

The WMF_testsystem is one of the MediaWiki instances in the Wikiation Extension Testing Environment. This system is intended to mimick the English Wikipedia.

The way this configuration is build is distinctly different from the English Wikipedia. This system is scratch installed, it does not share the files needed for CentralAuth and MergeAccount, the Cite and the SpecialCite are in separate folders and last but not least, there are a few extensions that I have not figured out how to install yet.

What I really liked was that it took me only five minutes to rebuild my environment once Wikipedia was updated. Three minutes to change my installation script and two minutes to run it.
Thanks,
GerardM

When less is more

When I enter the command ls revisions.mediawiki: I get a long list of revisions. They are in lifo format and it is a long list. When I enter this commond from within the Wikiation Installer in PuTTY, the list has a length that is too much for the PuTTY buffer. It is possible to limit the output by adding "limit 20". This works fine.

When you run the command from Bash, like this: ./installer.py ls revisions.mediawiki: it is possible to add command separated with a pipe. I tried the less and this worked fine for me. That is until I learned that I could break the pipe by pressing the "q" to quit, that gave a traceback.

The good news is that we can inform you what revisions for MediaWiki are available. It is possible to limit the number of most recent revisions, it is even possible to use bash for this and the pipe will get some more plumming.
Thanks,
         GerardM

Proof of the pudding

One of the objectives of the Wikiation Extension Testing Environment is to have the ability to test the functionality of a specific configuration. A prerequisite is the ability to built an environment that has the same configuration.One of the missing components was the ability to install a specific revision of MediaWiki itself. The screen shot demonstrates our new ability to do this.

The way we install our extensions is different from doing it manually. We expect to be able to install and uninstall extensions and consequently each extension needs to be in its own directory. For the Cite and the Special:Cite extension it meant that they are split in seperate directories. They install successfully and do not break MediaWiki so we assume that this is fine.

At this moment we assume that things are fine when the system installs and when we can save a page. Running tests will be our goal once we can reliably build our testing environment.
Thanks,
GerardM

Installng Metavid

As we are developing and testing the Wikiation Installer, we are recreating  relevant configurations. Metavid is the open video archive of the US Congress and the Metavid software will be the basis of video support for MediaWiki. People are interested in learning about these software developments and being able to build such environment is fun.

The metavid profile brought several things to me that were new. There are programs external to MediaWiki, programs that are not part of the LAMP stack that are prerequistites. We do not want to install such programs from within the installer but we do want to be able to test for such prerequistites. At this stage, we do not test for this, we leave this on our "wish list". 

I had to install a SQL script in order to install metavid tables in the MediaWiki database. This was beyond me. Kim helped me out and now I have an example that I can copy. Now that the files install, I also needed a way to know that the files had installed. Michael Dale, the developer of Metavid suggested to use phpmyadmin. This was an attractive suggestion but, it does add to the clutter in what should be a clean environment. 

What to do.. With hindsight the answer was obvious; one of the tests checks if new tables have been added to the database. So eating our own dogfood was in order. The isolation check was revived and indeed tables proved to have been added for Metavid.

I may have done a good job. I do not know as I do not how to test the functionality of this metavid environment. 

Thanks,

       GerardM

Revisions and Tags

When you are installing MediaWiki or MediaWiki extensions, it is often crucial to be able to install a particular revision or for a particular "tag". I am quite pleased to announce that the Wikiation installer is now able to install both for specific revisions or tags.

This functionality is currently very much bleeding edge. I do not know yet how to apply the tags to the scripts that I already wrote. So I will do some more documentation and some other work...
Thanks,
        GerardM

How to install RSS_Reader ?

RSS_Reader is an extension. It must be because it says so on the MetavidWiki and on the MediaWiki wiki. The problem that I face is that it has not been packaged as an extension.

That is a shame for several reasons.

• I cannot install it
• messages do not get internationalised and localised at translatewiki.net
• it prevents me from building a complete MetavidWiki test wiki with the installer
  

The information on the MetavidWiki says that it is "version 0.2.3". Now what if I need to run the software on a previous release. I may need "version 0.2.2" and I cannot get it because it is not in SVN.

I confess that I am not a developer. But I am quite sure that when you follow the instructions, you have something that might actually work. A developer, a real one, someone who Brion trust to submit to SVN, could then make it an extension so that I could do my "monkey see, monkey do" routine.
Thanks,
      GerardM

Installing the UNIWIKI extensions

When you want to test software, you need to be able to set up an environment quickly and repeatedly. The Uniwiki extensions are tested by customers of Wikiation, a programmer is working on the code so it makes sense to be able to create this environment quickly.

Programming in bash is something that I did a really long time ago, so it was hard work for me to get the install scripts right. Once I had the first one working, the other nine were easy.

I am really fond of the Configure extension so I always install that one. The only thing that I have done is install the software and I leave it to you to test it.
Thanks,
GerardM

Documentation ....

Documentation is almost universally something that you may do at the end. As such it is quite similar to testing. Python has this nice little tool called pydoc. It helps you generate documentation as you are working the code and as a consequence it provides a great incentive to work on documentation from the start.

One of the nice features is a web front end to the documentation. If you are interested, have a look at our documentation. This is the documentation of the bleeding edge, so you may see things that have not even been committed to SVN. To get an updated to the latest version in SVN, use update_self from within the installer.
Thanks,
GerardM

Installing a MediaWiki system

How long does it take to install a new MediaWiki system with Semantic MediaWiki?  It takes me a minute and yes, I cheated because I automated the process. This kind of automation is extremely important to us because it is the basis for running automated tests.

Thanks,

       GerardM

Installing SemanticForms

I created an installer script for SemanticForms, everything should work but it did not. I could not find what the problem was so I asked MinuteElectron to help me out. After some research he found that it had to do with the order in which the SemanticMediaWiki and the SemanticForms were loaded; they are loaded in alphabetical order so the SemanticForms came in first.

I have created a hack to get around this; I have renamed the SemanticForms.settings.php to xSemanticForms.settings.php. The documentation is clear; Semantic MediaWiki has to be installed and in this way MediaWiki is aware of this. Adding an x does the trick.

I wonder if this is one of those temporary solutions that will prove to be permanent.
Thanks,
     GerardM

Installing SemanticResultFormats

Programming is what I used to do a lot. The reason why I am programming again is because I am working on the documentation for the Wikiation Installer. I cannot document properly what I do not really understand and programming some installer scripts is one way of getting to grips with the issues.

I am testing in the "Fosdem" environment, this is where developers may test the existing functionality... I did install Semantic MediaWiki and I decided that I wanted to install SemanticResultFormats as well. I read the documentation and I found that due to a non standard include, I had to write an installer script. This is annoying but trivial

I tested it, the version special page reports it as being installed so I was good. However, it struck me that SemanticResultFormats has in SemanticMediaWiki a prerequisite. So I dabbled some more in the download.sh only to come to the conclusion that this is not the place where prerequisites are to be tested. The reason for this is that the include would be excercised anyway. When SemanticMediaWiki is not there, the whole installation should abort.

Writing installation scripts does help me understand the environment, it will help me write better documentation and in the mean time, I find where the software can use some more refinenment. I do confess that I am looking for the corner cases and so far I am really pleased how the installer is improving.
Thanks,
GerardM