OpenLink

From MetaWM

Contents 1 WARNING 2 Rename? 3 Overview 3.1 Aim 3.2 Why 3.3 How 3.4 Example workflow 4 Features 5 Design 5.1 Interfaces 5.2 Components 5.3 Packages 5.4 Consuming an interface (Hello World) 6 Running 7 Download

WARNING

This is very much work in progress and this is currently not a very coherent description of what OpenBuild does, why it's needed or how to use it.

Having said that I welcome contributions or even if you can send me your questions that would be helpfull.

Rename?

This project has been renamed from "OpenBuild" to "OpenLink". Even this is just a placeholder name for now, but I think OpenBuild was missleading since it gave the impression that the tool was comparable to build tools such as Scons/autotools etc.

Overview

Aim

OpenLink aims to provide a portable framework for highly modular programming projects. Primarily that involves:

• automating the task of choosing the optimal link strategy for binaries.
• providing a consistent interface to developers that might be targetting different platforms, such as for:
  • APIs relating to the access of binary objects.
  • generating packages that can be distributed to end users.

It is a developer tool, intended for projects written primarily in C.

A "link strategy" in this context refers to how a developer decides to link the different components of their project together.

E.g. Consider a project that provides a large number of small tools (imagine Gnu coreutils) and that there is some overlap between them over basic functionalities they require such as linked list implementations and hash algorithms etc.

OpenLink will work to answer the following:

• What code can be placed into shared libraries?
• Should anything be loaded dynamically at runtime as/when it's needed?
• Should the tools have seperate binaries or should we link everything into one monolithic binary and use other means to choose the tools (like busybox.)
• What are the trade offs, and considering the developers priorities what it the best layout and linking strategy.

Why

This project has come about through my work on MetaWM, which aims to be a modular framework for window manager developers. OpenLink will be the tool used to bring all the required modules together for a particular window manager.

How

There are a few key problems to implementing OpenLink

1. There isn't a suitable C programming model yet that solves these issues:
   • C doesn't have namespacing of exported symbols so "highly modular" systems have to consider ways of eliminating symbol clashes.
   • We need to write code in C that has dependencies on external functionality but lets you remain agnostic of the final linking strategy.
     • Notably Gnu's libtool does help by providing libltdl which implements a portable dlopen() API that can even support platforms with no suitable dynamic linking API, but we need to suplement it a bit for our end.
2. There arn't universally agreed uppon tools for compiling and linking C code.
3. There isn't a universally accepted packaging format.

CONTINUE HERE -- 23/5/2006 - I paused here on my task to update this documentation

Example workflow

Interfaces 

Define a contract that modules must adhere too. Interface development involves declaring (in XML) the function prototypes that the module will provide. Idealy it should also involve writting a detailed textual contract describing the functional requirements of the module.

Components 

These refer to a single specific implementation of an interface contract.

Packages 

These are - user significant - collections of components grouped together into a distributable archive suitable for a particular audience. For example if your audience is generally Linux developers/competent users, then OpenBuild will create you a source package based on autoconf, automake libtool etc. Alternativly a package for Windows users could instead give you a wizard based binary package. Packages are also defined in XML.

Features

1. Basicly it just provides a way for you to group small pieces of code together to make a bigger more useful blob of code.
2. Portable packages. For example OpenBuild should be able to generate packages targeted at an OS's without any dynamic loading facilaties.
   • Note: OpenBuild's own portability is currently at least bound by Python portabilty, but since it is intended to be a Developers tool I consider this a non issue.
3. Minimal overheads. For example I wanted to be able to link modules statically to allow pedantic binary size and startup (linkage) optimisations.
4. Namespace mangement

   C doesn't have special namespacing features, instead developers usually prefix symbol names or use OS specific facilities such as dynamic loading to seperate namespaces.

   
   

   The difficulty with prefixes is if they are too long they become a burdon to type. If they are too short you can quickly exhast the meaningfull prefixes and it can become awkward when you want to allow many implementations of a single interface.

   
   

   The difficulty with dynamic loading facilities is that they arn't standardised accross platforms and some platforms don't even have such facilities.

   • OpenBuild aims for 100% assurance that you won't come across namespace collisions, even for projects that could have hundreds or thousands of independently authored modules.
   • You can easily use multiple implementations of the same interface together.
5. OS specific dynamic loading APIs are abstracted.
   • OpenBuild is responsible for dealing with linkage issues. The apis are designed to abstract from the OS and linkage methods so OpenBuild can e.g. choose to statically link half of the projects code, but leave the more specialist code as dynamic modules. (To save runtime memory)
   • It's designed to be able to take statistical usage data as input to its linkage descisions. This may become more important for huge projects with thousands of modules.
     • In this case startup lag due to dynamic linking may be noticable
     • The modules that are rarely used can be left as dynamic modules.
       • Going further you may have large groups of modules that may be rarely used but are used together, these can be linked together e.g. into a larger object.
     • I have also considered the possability of logging enough link information so that by gathering new statisics you can ask OpenBuild to re-link the software (Note: that in this case OpenBuild also becomes a tool for end users) (At this point I was getting ahead of myself and imagining a whole operating system built using OpenBuild and being able to optimise linking of the whole system.)
     • The current plan (fairly long term) is to have a special build method that enables statistical profilling of module usage for use as input to the linking descisions.

Design

An OpenBuild project is defined in terms of "interfaces", "components" and "projects".

Interfaces

are implementation independent and describe a contract for a piece of code to adhear to.

Components

correspond to a single implementation of a single interface.

Packages are

meaningfull (to end users) collections of components used for distribution.

Each of these is described using XML.

Interfaces

Here is an example interface.xml

I'd have to check my notes but I believe the language tag doesn't enforce that the implementation be written in C, but rather the schema used to describe the interface.

i.e. It's quite a dynamic XML schema (which I imagin will make validation a bit awkward)

I felt it was more important to have a schema that matched well to the anticipated language of development and was intuative to the developer having to write the description.

Eventually I want OpenBuild to be able to do cross language linkage, but I didn't think it was best to enforce the compatability at this level by e.g. having a unified interface description language, as it would get in the way of the more common development.

<interface>

   <namespace>org_sixbynine_test_eg__0_0</namespace>
   
   <language>C</language>

   <dependancy>
       <system_header>glib.h</system_header>
       <system_package>glib-2.0</system_package> 
   </dependancy>

   <function>
       <name>test</name>
       <arg>
           <name>message</name>
           <pointer>
               <type>unsigned char</type>
           </pointer>
       </arg>
   </function>
</interface>

Components

Example component.xml

<component>

       <namespace>org_sixbynine_test__0_0_0</namespace>

       <implements>org_sixbynine_test_eg__0_0</implements>
       
       <type>source</type>
       
       <language>c</language>
       
       <dependancy>
               <namespace>org_sixbynine_core_wm__0_0</namespace>
       </dependancy>

       <dependancy>
               <system_package>glib-2.0</system_package>
       </dependancy>

       <source>
               <location>./main.c</location>
       </source>
       
</component>

Packages

Example package.xml

<package>
       <name>test</name>
       <version>0.0.0</version>

       <component>
               <location>components/org_sixbynine_test__0_0_0</location>
               <program>test</program>
       </component>
       
       <extra>
               <file>design.sxw</file>
               <location>doc</location>
       </extra>

</package>

Consuming an interface (Hello World)

#include <stdio.h>

#include <org_sixbynine_test_eg__0_0.h>
#include <openbuild_interfaces__0_0.h>

int
main(int argc, char **argv)
{
   int handle;

   handle=ob_take(ORG_SIXBYNINE_TEST_EG__0_0, OB_FIRST_IMPLEMENTATION);

   if(ob_initialised(ORG_SIXBYNINE_TEST_EG__0_0, handle)){
       // This shows of the namespacing with 4 calls to the same function...
       test("Hello,");
       eg_test(" World!\n")

       eg__0_0_test("Bye,");
       org_sixbynine_test_eg__0_0_test("World!\n");
   }else{
       fprintf(stderr, "failed to initialise test component\n");
   }

   ob_release(ORG_SIXBYNINE_TEST_EG__0_0, handle);

   return 0;
}

Running

Currently the main script is gen_package.py

You simply pass it a package.xml filename and it will build you a package with autotool build scripts.

Currently OpenBuild has one test package, but it depends on a component from my MetaWM project.

assuming you have the MetaWM project checked out at ~/src/metawm and openbuild checked out at ~/src/openbuild then you should create a ~/.openbuildrc.xml file as follows...

<config>
   <repository>
      <path>/home/USER/src/openbuild/tests</path>
   </repository>
   <repository>
      <path>/home/USER/src/metawm</path>
   </repository>
</config>

You should add ~/src/openbuild/bin to your $PATH and then you can run:

gen_package.py tests/packages/test_package.xml

This might create an autotooled project under a directory called test.

Download

You can browse my Subversion repository here

To check out a copy:

svn co http://svn.sixbynine.org/openlink/trunk openlink

or if that is broken there is a "stable" branch...

svn co http://svn.sixbynine.org/openlink/branches/stable openlink_stable