April 7, 2006 19:12

Building OpenOffice.org, too fast-too furious

Building OpenOffice.org, too fast-too furious

% of build time for OpenOffice.org 2.0.2

It has been more than a year now that i have been building OpenOffice.org, after all this time I have even considered printing my own T-shirts with the text: I build OpenOffice.org, do you?. After spending more than 500 hours building the software (yes, it is sad becouse it is like 20 full-days), i always wondered: where all this time goes?. I finally got the time to figure it out.

When OpenOffice.org is building, you will find the following type of messages:

[..]
=============
Building project helpcontent2
=============
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/helpcontent2/source/auxiliary
mkout -- version: 1.6
[...]

So, what I did was to modify the Perl script solenv/build.pl that manages that process as follows:

     print $echo . "=============\n";
-    print $echo . $text;
+    $timestamp = time();
+    print $echo . $text . $timestamp ."\n";
     print $echo . "=============\n";
 };
[..]
=============
Building project helpcontent2: 1144089629
=============
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/helpcontent2/source/auxiliary
mkout -- version: 1.6
[...]

I collected all the timestaps (seconds since 00:00:00 1970-01-01 UTC) and reached some interesting conclusions. OpenOffice.org (in my build) calls 163 different modules during the build. Most of the functionality for the user is in a few top level projects: sw (Writer), sc (Calc), sd (Impress and Draw) and dbaccess (database). Functionality that is common to all applications is in the desktop, framework, svtools, svx and sfx2 projects. 20 modules take 75% of the compilation time (see next list), helpcontent2 takes almost 1/3 of all the compilation.

136__helpcontent2     28,44%
105__officecfg	      6,47%
164__instsetoo_native 5,52%
161__sw	              5,06%
147__svx	      4,62%
156__sc	              4,55%
157__sd	              3,21%
14__offapi	      2,52%
22__transex3	      2,01%
150__dbaccess	      1,74%
122__svtools	      1,60%
29__icu	              1,55%
123__connectivity     1,45%
137__xmloff	      1,34%
140__sfx2	      1,20%
97__i18npool	      1,20%
155__sch	      1,18%
153__filter	      1,09%
116__vcl	      1,07%
   __others (144)    24,18%

This list is an extract of the results, it shows the 20 top modules and their compilation time, the number NN__module indicates the order it is compiled. For example, if we take

164__instsetoo_native 5,52%

This indicates that the process of packing the software takes 5% of compilation time. Why this information is usefull?

In the Sun X4100, a full compile using ccache for one language (i tested Swahili) sums up to 87 minutes. So, the first conclusion is that rebuild helpcontent2 is CPU expensive!.

Two types of process are infamous in the building:

java -Xms256m -Xmx256m -Djava.library.path=/lib:/lib64:/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64 
-cp.:../../unxlngi6.pro/class:/home/mbuild/java/j2sdk1.4.2/jre/lib/rt.jar:.:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/xml-apis.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/xercesImpl.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/xt.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/unoil.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/ridl.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/jurt.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/jut.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/xmlsearch.jar:
/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/solver/680/unxlngi6.pro/bin/xmlhelp.jar:
/usr/share/java/db.jar com.sun.star.help.HelpLinker @/tmp/mkGWSrlH

and a sed section:

/home/mbuild/ooo-build-2.0.2.2_20060328_build1/build/OOO_2_0_2/helpcontent2/source/text/shared/01
dmake: Executing shell macro: +echo $(IDLPACKAGE) | $(SED) 's/\\/\//g'
-------------
..............................................................................................
..............................................................................................
..............................................................................................
done

Rebuilding OOo after updating a localization

I have been also measuring what is the time that requires to rebuild OOo once a localization has been updated. I compiled OOo with ccache and using the --with-lang=ALL flag. Over 70+ languages where compiled in a couple of days leaving a C/C++ compiler cache of over 1 GB.

My first conclusions are:

  • Rebuild OOo takes < 10 mins once a localization has been updated if helpcontent2 is not modified.
  • Rebuild OOo takes 20 mins < t < 30 mins if few updates are present in the helpcontent2 module.

It seems a good approach to find easy ways to handle the help independently of the general build process. This is something to investigate! Hints?