Group Enabled Cluster Compiler: gecc

NEWS

20021101: a many smaller bugs fixed and unit tests expanded

20021019: some bugfixes

20021014: implemented a queue for jobs, that could not be dispatched immediately. implemented (the beginning) of unregistering of compile nodes. Now using UnixDomainSockets for gecc to geccd communication.

20021012: geccmon, a monitoring tool added. it's python and uses curses.

gecc monitor (c) by j.beyer@web.de jobs: | 0/7 host | file | size | time [sec] | speed [kb/sec] ----------+-------+--------+------------+--------------- dilbert| 10 | 1 m | 8.63 | 127.15 asok| 1 | 113 k | 3.44 | 33.79 alice| 6 | 564 k | 3.05 | 189.44 ----------+-------+--------+------------+--------------- *** this is work in progress *** please visit http://gecc.sf.net/

20021007: Big Bugfixes: the return code of the compiler was not propagted correctly, thus many configure test gave errors. Big Features: the compile job dispatcher now comes in two flawors: round robin and fastest again. It's chosen by a commandline switch. There is also a job limit per machine.

20021006: reworked the transport layer and fixed some bugs.

20020929: Bugfix release: gecc handles now errors in the precompilation phase (e.g. #error) and output targets like "-o dir/file.o".

20020928: The combination of caching and distribution is working. With this release it is the first time to have both features working with the much simplified process model. There is still work to do, but with this release gecc could be used to compile working software, as it is tested with several mainstream packages as found on sourceforge.

see also the ChangeLog.

milestone: mozilla builds on 3 machines (and runs)

What is it?

a tool to speed up compilation of C/C++ sources. It distributes the compilation on a cluster of compilation nodes. It also caches the object files to save some unneeded work.

How does it work?

gecc is a compiler driver. The compilation from a file.c to a file.o could be separated into two steps:

• preprocessing, from file.c to a file.i
• compilation, from file.i to a file.o

The preprocessing step depends on the input source code (incl. all used header files), the comandline and the version of the used compiler. If you recompile a project without changes then you dont have to recompile the object file, but could lookup their object files in a cache. gecc has implemented such a cache. If you are more than one developer, working on a large project then there is a certain probability that a few source files are the same in your and your friends working directory. The resulting object files should be the same, anyway who compiled them (assuming all other parameters are equal). You either could compile them on your own, or take the object file from the cache, where your friend put his object files. The second choice is faster.

As said before: the object code depends on the source, the comandline options and the compiler on a certain machine. gecc makes these 3 values (or a hash of them) a key of a cache entry. The object file is the cache entry.

The second step, the compilation of the preprocessed source to an object file, could be distributed on other machines without the need of equal headers or libs installed on both machines. gecc implements a distributed compilation, this gives an extra speed up.

Where could it be useful?

• You are on a single machine and recompile sources, often with little or no changes (e.g. building rpms and fiddiling aroung with the spec file is such a case). Here you could cache the object files to speed up the compilation.
• You are working with more than one developer on one (reference) machine. They could use the compilation work done by others.
• You are on a slow machine and want other machines to take part in the compilation. Maybe only on the old machine are the correct headers/libs installed and you can't simply do all the compilation on the faster machines.
• Cross compilation: I have a nice old sparc here and it takes ages to compile a new kernel. I plan to setup a cross compiler on my faster machines and let them help the old sparc.
• Notebooks: this is a similar case to the example above. If you sit in your garden and are connected via wlan to your other machines, then they could help compile.
• If you are working in a group of developers, then the distributed compilation help to better utilize your CPUs, this saves time and money.

What is the state of the project?

gecc is a proof of concept. It is heavily inspired by ccache and distcc. You could chain these tools to achieve the same goals gecc tries to reach. Both tools are much more mature and work in production environments. gecc just started with a little different concept. gecc has a central component (distcc has not).

My idea is that gecc could better handle a varying set of compile nodes: if you have some machines that only from time to time could help in distributed compiling than this is nice.

With a central component it might be easier to monitor the compilation and distribute the compile jobs.

Right now gecc is only useful if you read the source. I will add testcases and instructions on how to use it in a few days.

Cached compilation and distribtution work in principle, but errors are not detected in many case. I believe in "release early, release often", so I make the source available now. Feel free to contact me with any feedback

If you test gecc yourself, then I would like to hear from you. I am interessted in bug reports as well as "works for me". I list some of the packages that I or some kind soul has compiled distributed and cached here.

Download

Right now I would suggest that you use cvs, as described on sourceforge

But for your convenience, here is a snapshot.

visit also gecc on sourceforge.

Contact

Mail me: j.beyer@web.de or post on the mailing list