Player in Clutter: continuous integration

Ideas and proposal for setting up continuous integration (jhbuild, buildbot and a correctness and performance testsuite) in the player built with Clutter.

Translation of the original in Spanish. Read it in: CatalàCastellano

Part 7 of 7Clutter notes
  1. Clutter concepts
  2. Notes on Clutter
  3. Installing Clutter with jhbuild
  4. Clutter snippets: D-Bus, GStreamer and ClutterScript
  5. Some notes on Clutter development
  6. Player in Clutter and DBUS
  7. Player in Clutter: continuous integration

This page was on the oriolrius.cat wiki (2009–2012). I bring it back here as it was.

Background

The goal of continuous integration is to continuously track the changes in a project's code, in order to detect faults or problems at the moment they are made, that is, to achieve fast error detection that avoids later problems. In the end it is a matter of quality control.

To carry out continuous integration of a project, it must meet a series of requirements:

  • Use of version control. This way changes in the code can be detected easily, as well as which version is the one that generates problems
  • Automatic build. The project must be buildable with a single command.
  • Complete testsuite. The project must have a set of tests that will determine whether there is some kind of error in the code

There is also a series of recommendations:

  • Regular commits. This prevents the different development branches from diverging and makes it take longer to detect problems
  • Every commit should be built and tested
  • The build should be fast. The sooner the code is built, the sooner faults are detected
  • Tests should be run on a clone of the production machine.
  • Easy obtaining of the latest versions, so that it is easier to test and therefore to detect problems
  • All developers should be able to see the test results. This way, if there is any problem, everyone will be able to see it easily.

Goals

The goals to be achieved using continuous integration are:

  • Check the correctness of the software
  • Check the performance of the software
  • That the cost of implementing the system is low.

Ideas/proposal

Taking into account what has been said in the previous points, in order to implement a continuous integration system the following tasks would have to be carried out:

  • Create a software build system
  • Implement a continuous integration server
  • Develop a complete testsuite, with both correctness and performance tests.

Let's go deeper into each of the points

Software build system

Taking into account that one of the fundamental goals is performance evaluation and that the player is based on clutter and that a large part of the performance will be determined by this library, it may be interesting to also test the software with different versions of clutter. Clutter uses jhbuild as its software build system, so the same system could be used to build the whole software stack

Continuous integration server

There are a multitude of applications for carrying out this task (see http://en.wikipedia.org/wiki/Continuous_integration). However, taking into account the choice of build system indicated in the previous point, we are interested in some system that integrates easily with jhbuild. jhbuild is part of the Gnome project, which is using a continuous integration system (buildbot http://buildbot.net/) and they have scripts developed for integration with jhbuild (http://live.gnome.org/BuildBrigade https://www.ohloh.net/p/jhbuild-buildbot-scripts). That is why I think we could use this system.

In addition, there are a couple of less relevant questions that I consider positive: jhbuild and buildbot are both developed in python, so all the tools use the same programming language, and behind the integration of jhbuild and buildbot are the people from igalia, which implies a certain geographical and idiomatic 'closeness'.

Testsuite

Correctness

Python offers a framework in the standard library for creating tests: http://docs.python.org/library/unittest.html

Performance

Apart from taking into account issues relating to performance in python http://wiki.python.org/moin/PythonSpeed/PerformanceTips in order to run performance tests. For this there are several options:

I have also found an interesting article by Andy Wingo on how to improve memory consumption in python: http://wingolog.org/archives/2007/11/27/reducing-the-footprint-of-python-applications

The problem with all these solutions is that they are either designed to debug a program (not to track it over time) or they offer limited information. In short, python is not a language that allows much introspection in terms of time/memory consumption. The most reasonable thing would be to use a combination of the pyunitperf framework with timeit, however, it must be taken into account that this will only give us information about the real time it takes the application to perform certain tasks.

Tasks to be done

  1. Create a basic performance testsuite for the application.
  2. Validate whether the information we can obtain from the testsuite seems interesting to us for implementing a continuous integration system
  3. Implement/test buildbot
  4. Connect the results of the basic testsuite with buildbot
  5. Create a jhbuild module for the player
  6. Implement the continuous integration system on a machine equivalent to the production one
  7. Create several environments with different versions of clutter/pyclutter (stable, unstable, etc.) to be able to evaluate whether a change in performance is due to clutter or to the player
  8. Complete the testsuite for the player
  9. Implement