Gnome and Fedora-12 on ARM

Here are some screenshots of my kirkwood DB (1.2 GHz and 512 MB RAM, Matrox G550 graphics card) running Gnome. Pretty neat!
I am going to try XFCE desktop over the weekend. Lets see how that goes...

Here is a picture of my setup (taken 6 months ago)
(click to enlarge)

Another awesome Fedora release!

Fedora-ARM 12 is available!

Fedora-ARM 12 is now available.

The RFS is available at:
http://ftp.linux.org.uk/pub/linux/arm/fedora/rootfs/rootfs-f12.tar.bz2

The following package groups are available: Base, Core, Base-X, GNOME-Desktop, XFCE-Desktop, Java, Java-Development, Admin-Tools, System-Tools, Web Server, and commonly used embedded packages.

As usual it is built for ARMv5 EABI, soft-float, little endian.

For more details head to:
http://fedoraproject.org/wiki/Architectures/ARM

kernel and Fedora-ARM

Handling the kernel package on ARM is different than other architectures. Our kernel package typically provides only the kernel-headers. This is because, there are hundreds of ARM boards out there and everyone will use their own optimised and stripped down kernels anyway!

The side-effect of this choice is that there is no kernel package in the rpm database and hence, the packages that Require kernel (about 15 of them) won't have their dependencies satisfied. (Some examples would be, fuse, xorg-x11-drv-ati, pulseaudio, libdrm, etc) Currently, we work around this by changing the Requires on the kernel by Conflicts. Here is a sample patch. (It is obviously not a very elegant solution and any suggestions are welcome)

Now, this isn't much of a problem for packages that just Require kernel. But, now xorg-x11-drv-nouveau Requires kernel-drm-nouveau. It is rare for any ARM kernel to be compiled with nouveau support, so it makes less sense to change this Requires to Conflicts. Ofcourse, we can assume that whoever is trying to use a nVidia card with an ARM board knows what the hell he's doing, but the solution doesn't seem right anyway.

Any suggestions on how this can be handled?

Fedora-ARM: alignment errors

Alignment Errors:
We usually come across a lot of alignment error problems while building packages for Fedora-ARM.

Here is a quick example of the problem:

#define SIZE 20
#define CHARVAL 'a'
#define INTVAL 0x0a0b0c0d
main()
{
     char buffer[SIZE] = { 0, };
     char *p = buffer, c;

     printf("c value = 0x%x\n", CHARVAL);
     printf("i value = 0x%.8x %d\n", INTVAL, INTVAL);

     /* Ok, lets write them in the buffer one after the other */
     *p = CHARVAL;
     p++;
     *(int *)p = INTVAL;

     /* So the buffer should be 0x610d0c0b0a */
     print_hex_string(buffer);

}

The output of this program on x86 machine:

c value = 0x61
i value = 0x0a0b0c0d 168496141
0x610d0c0b0a000000000000000000000000000000
^^^^----------------here

As expected :-)

And on ARM?

c value = 0x61
i value = 0x0a0b0c0d 168496141
0x0d0c0b0a00000000000000000000000000000000
^^^^----------------here

And there goes your value of "c" stored in the array.

These alignment errors are often seen on the ARM architecture. Basically it expects a word to be on a word boundary (last 2 bits should be zero). And if it isn't, well expect the unexpected. As can be seen above the value of c has been completely over-written.

The same goes for pointers as well.

/proc/cpu/aligment:
This file displays the list of alignment errors that have been encountered thus far.

You can echo various flags in this file to change the kernel's behavior when an alignment error is encountered.

• 0: ignore
• 1: warn (/var/log/messages)
• 2: fixup
• 3: signal (core dump for analysis)

For a detailed overview: http://lecs.cs.ucla.edu/wiki/index.php/XScale_alignment

Lots of optimisations

While mass-building for F-11, we realised that there are a lot of areas we can work on to make the build process faster. ARM boards are not as powerful as the i686 machines, so, the actual build time cannot be reduced beyond a certain limit. But, we can increase the throughput. So here's what we've been working on for some days.

1) We got some new builders in the build-pool. Each one has 2 GB RAM and 1200 bogoMIPS processor. So, now we have 8 builders total in the build pool. We hope to add 2 more in some time. With 10 builders, the mass-builds would be much much faster.

2) Eliminate buildSRPMfromSCM tasks. Frankly, they were more of a red-tape. buildSRPMfromSCM is already done on the fedoraproject koji, it would be much much better (and faster) to just pick-up the SRPM and build. So, we wrote wrapper scripts to do just that. This has been of great great help because the boards are always doing some "meaningful" work now.

3) Optimise postgresql database. We vacuumed (fully) the database and reindexed it and it is blazing fast.

4) Also, we have scripts to partially automate analyzing build failures, determining which library sonames have changed upstream so that we need to build them first, etc. The build process is now fairly defined and streamlined.

And we have already started working on the base-set of F-12!

Fedora 11 Status

Since the last announcement of F-11 packages, a large number of packages have further been built. There currently are above 9000 rpms in the Fedora-11 repositories. Our effort has been to make available the following package groups:

Base
Core
Web-server
Gnome Desktop
Xfce desktop
Development Tools
X server
Web development
Admin Tools
System Tools
Java

The Fedora-11 updates for a large set of packages are also now available.

For Fedora-12, we would like to start tracking packages preferably when all the shared lib .so version updates are settled for f12 (thus avoiding any additional rebuild cycles at our end). Most likely this will happen by the Fedora-12 Beta Freeze date (29 Sept, in 3/4 days).

dhclient, firstboot and hald woes ...

In the last 2-3 weeks, I have run into some issues on my ARM board.

1) The first one was the udev-dhclient issue. The basic rootfs from the Fedora-ARM page worked fine for me. But, when I installed some packages, the board refused to reboot. It used to freeze on "Starting udev". After tinkering around a lot with udev, I finally decided debugging udev wasn't my thing. So instead tried to determine which package was causing the problem. After some brute-force and binary search, realised that dhclient was the culprit. After installing dhclient, the board hung at "Starting udev" on the next reboot. Weird. For now, I have just refrained from installing dhclient and everything is working fine. Any one have any ideas about this? (Probably dhclient gets the lease calculation wrong because udev accesses the hardware clock at the same time?)

2) Now firstboot. Its weirder! So, after I installed X and Gnome package group and restarted the board, it used to hang on the login prompt (run level 3). I could key in characters using the keyboard, but hitting return gave no response whatsoever. I couldn't even ssh into the board. At first, I thought it was a system freeze, but I enabled autologin on tty3 and wrote an infinite loop in .bash_profile and it worked fine. Finally, after 1-2 days of wandering off in different directions, the only option that remained was: the keyboard input is losing its way somewhere. So, copied my i386 system's /etc/sysconfig/keyboard to ARM's rootfs. And the system now responded fine to the keyboard input. But, it hung on "password:" prompt. Damn! Finally, realised that I need to run firstboot, so that it configures keyboard and authentication (using pam.d). (Now, as I didn't have a console on this rfs, took the basic working rfs, ran firstboot there and copied the generated configs on the new rfs.) Now things worked fine! firstboot!

3) hald. I still havn't been able to resolve this one. Its been 3 days and I have no headway. Problem is, whenever I start hald, the system just freezes. No I/O, no ssh nothing. A quick infinte loop in .bash_profile suggests that the system has crashed. I have posted the logs and lshal outputs to the mailing lists. Waiting for a reply. Without HAL, my USB mouse and keyboard aren't working in Gnome.

Do leave a message if you face a problem trying something out with your setup ...

Steps for getting X-Gnome working...

1) Make sure you have your kernel compiled with atleast the following options enabled:

• Device Drivers -> Graphics support -> Direct Rendering Manager
• Device Drivers -> Graphics support -> Display Device Support
• Device Drivers -> Graphics support ->
• PCI (graphics card is usually on the PCI bus)

2) With my card, I had to add the following kernel commandline so that the driver initialised my video card at init : "video=matroxfb:init,vesa:0x11B" (I have a Matrox G550 card)

With the above two steps done, you should be able to see a console on the monitor.

3) Now, install the Gnome and X server package groups (atleast the packages listed under "mandatory".
Gnome:
PolicyKit-gnome
control-center
fedora-gnome-theme
gnome-applets
gnome-panel
gnome-session
gnome-terminal
gvfs-fuse
metacity
nautilus
yelp
gdm

X:
bitmap-fonts
desktop-backgrounds-basic
xorg-x11-drivers
xorg-x11-server-Xorg
xorg-x11-xauth
xorg-x11-xinit
)


4) Now, install the "system-config-display" package and run "system-config-display --reconfig" as root to generate xorg.conf. It generates this configuration file as some temporary file, so, copy it to /etc/X11/xorg.conf. It usually contains only generic stuff. So you might want to edit it. Eg. My xorg.conf was trying to use the fbdev driver (since, I had Matrox's fbdev support enabled in the kernel). So, I edited Xorg.conf to use the "mga" driver (Matrox driver). etc

5) Install "firstboot" and run it manually so that it configures your keyboard and authentication. Optionally add a user using useradd. Make sure that /etc/sysconfig/keyboard and /etc/pam.d/* files are generated.

6) Now just run startx or init 5. Things should just work.
(you'd probably have to start HAL daemon manually and tinker with stuff here and there. But, this is the essence of what is required to get X up)

Also, you might want to edit inittab so that the next you reboot, you are presented with a graphical interface.

Let me know if you succeed in getting it working or have some issues :)

Fedora-ARM goes graphical ...

A picture speaks more than a 1000 words... Here is a screenshot ...

And here is the setup:
The graphics card is Matrox 550 attached to the PCI-E slot of the Kirkwood DB board.
The packages required for getting X and Gnome up aren't yet in the repos. Hopefully, I'll mash out a repo by the end of this week so that you can try it out on your boards!
yay!

Java Bootstrap:Part 4

Ok, the sinjdoc/xml-commons-apis dependencies is causing a real problem in build packages. Let me rebuild ant with bootstrap, and include these two in its install dependencies. Done.

Built packages antlr, jakarta-oro in koji.

Manyally built packages werken-xpath and wsdl4j in my setup (they needed xalan-j2 and ant-junit respectively) which is copied from x86 in my buildroot.

Ok, mx4j seems to be quite a critical packages, and its set of dependencies is huggeeee. I am just going the copy-x86 route with this package and build other packages. Built geronimo-specs manually based on this. It used the java-1.6.0 openjdk compiler (Just wondering whether it should have used java-1.5.0-gcj). I guess we'll find out...

Now built log4j which required jms (geronimo-specs)

Now velocity and bcel have a circular dependency. I am going to copy velocity build bcel, and then rebuild velocity. And additional there is avalon-logkit dependency in there. The avalon-logkit fails to compile. It gives up on some HTML syntax mentiond in a comment. Probably the 1.5.0 javac should be used here instead of the 1.6.0. I can't fire a build on koji yet as mx4j is not completely there. Let me copy teh jars and proceed with the bcel and velocity bootstrap.

built both of them. And now for xalan-j2.

Boyyy, I think eventually, for all the packages that I manually built, we should just run an automated script which bumps up the release sensibly (so as not to conflict with newer upstream releases) and rebuild.

xalan-j2 is now ready.
Build classpath-jaf. I can't still build avalon-logkit on koji because of missing mx4j. And in my chroot the build is failing.

Ok, in order to build mx4j, xjavadoc and xdoclet are the biggies in the sense of build dependencies. And xdoclet needs struts, which has further more deps... Lets first get to xjavadoc.
Built javacc, jrefactory deps. Now build xjavadoc. Done. Also build classpath-jaf and saxon in koji, all their buildreqs are now satisfied.

Now I plan to do a mass build of all the packages in the java-development group within dist-f11-build-javabs tag. Many of them will fail, but there will be a lot of successes I hope.

Java Bootstrap:Part 3

After about ~36 hours of build time, java-1.6.0-openjdk rpms were finally built :-) albeit in bootstrap mode.
http://arm.koji.fedoraproject.org/koji/buildinfo?buildID=13704

Trying to build avalon-framework. That needs avalon-logkit, which again needs avalon-framework. Pick the logkit jars from x86 and build avalon-framework.

Turns out the add-group-pkg thing that I had done with xml-commons-apis and sinjdoc (discussed in Part2) didn't go quite well. The way koji performs chroot setup is
1. install default build pkg group
2. init mock user by keeping mock uid/gid constant in the builder and the chroot.
3. install BuildRequires deps.

My adding these two packages in the default group, pulled in a few other packages which install user/group in the chroot in step 1 above. Thus in step 2 when mock tried to create mock uid/gid, the identifiers clashed with those already setup, thus failing the chroot setup itself.

Ok, so I took these packages out of the default package group, and included it explicitly in failing pkgs's BuildRequires.
With this I could build clean packages: java_cup, xml_commons_resolver, jlex, junit and regexp cleanly with only the BuildRequires included.

Now quite a few pkgs required jdom. It depends on xalan-j2. Copy from x86, and rebuild. worked.

Ok, now try to build classpathx-jaf. It failed with some error. Looks like the failed symbols are provided by classpathx-jaf itself. Copied the jars and trying to rebuild again. No luck.

There were some links to /etc/alternatives, that I was not copying which should have been done. Let me try that, probably that is why xerces-j2 was not getting built.

Great! that worked.

Somehow that didn't quite fix the problem of classpathx-jaf
Let me try classpathx-mail. That did work.

Now jakarta-commons-collections is needed by quite a few other things, but that BuildRequires ant-junit, which is something I don't have yet. I'll build with nodeps, lets see where it goes... Didn't quite work.. I thought so.. Hmm.. I'll probably have to rebuild ant such that it includes junit. But before that, lets see if the copying of contents of the x86 ant-junit rpm works... Did. Great!

Java Bootstrap: Part2

ant required xml-common-apis to be installed else it dumped something like this:
+ ant
/usr/bin/build-classpath: error: Could not find xml-commons-apis Java extension for this JVM
/usr/bin/build-classpath: error: Some specified jars were not found

So build xml-commons-apis, but that again uses ant, which needs xml-common-apis...

Hmm... If it is only looking for jars lets try to copy the jars from x86. I am assuming they are arch-agnostic.

# mkdir /tmp/a
# rpm -ivh xml-common-apis.rpm --root /tmp/a --nodeps --ignorearch
# cp /tmp/a/usr/share/java/* /usr/share/java -a

And try to rebuild xml-common-apis again.

Ok, that error went away, but I can't seem to build "javadoc" files. Some search tells me I need sinjdoc to build these. sinjdoc in turn depends on java_cup.

Build java_cup
But java_cup itself builds javadoc. Disable javadoc for now.
Build sinjdoc

Import these in koji.
Ok, now rebuild sinjdoc in koji, java_cup and xml-common-apis properly in koji without any patches. This can be tagged back to dist-f11.

Hmm... Now the bootstrapped "ant" doesn't get all the packages into its dependency list, particularly sinjdoc and xml-commons-apis, thus causing builds to break.

koji add-group-pkg dist-f11-build-javabs sinjdoc xml-commons-apis

That works...

Ok, now let me try to build java-openjdk. It has a lot of dependencies, but I am hoping all of them can be built using the gcjbootstrap option.

Build xml-commons-resolver with "nodeps" (it was looking for java-1.6.0-devel). This got built cleanly.

Ok, now let me try and compile java-1.6.0-openjdk with the current set. For xalan-2 and xerces-j2, I have copied the contents into /usr/share/java. There are a few additional dependencies like netbeans and jakarta-commons-logging, but the spec file says that these are needed only for visual-vm. Let me try to disable visualvm (there is an --enable-visualvm option passed) and see if openjdk gets built.

Java Bootstrap: Part1

This will be a document of the exact set of steps that I have taken for Java Bootstrapping.
(Thoughts,mistakes,curses all included...) So that I could look back at anything that I have done when something goes wrong later...

Bootstrap java-1.5.0-gcj
with bootstrap=1 option in the spec file.

Forcibly install java-1.5.0-gcj-devel (it has a missing deps on eclipse-ecj)

Build ecj with gcjbootstrap=1 (when it doesn't need ant) (because ant needs ecj again)

Now that you have ecj, build "ant". Ant still needs jaxp_transform_impl, but let me see what happens if I build with 'nodeps'. This is also being built with bootstrap=1. Sigh, the gcj is a really really time consuming task...

Woohoo, that is now done tooo :-) I hope all of this works later...


In the meanwhile, created a separate tag dist-f11-build-javabs in koji which inherits from the dist-f11-build-bs tag. All the rpms that have been "bootstrapped" built "with nodeps" etc. will be imported in this tag, thus not affecting the rest of the system. Once we reach a stage where rpms can be built OOB, then we'll put them in dist-f11. Imported all these rpms in that tag.

The gcc-java rpm

Wrote: /root/rpmbuild/RPMS/armv5tel/gcc-java-4.4.0-0.34.fa3.armv5tel.rpm :-)

That was good.. And now for the bootstrapping of the java subsystem packages. From what I hear it is HARD...!

irc.freenode.net:#fedora-arm

lennert> kedars: okay, so then java bootstrapping is next
lennert> kedars: that should be fun, with all the circular build deps
kedars> lennert: you bet, I guess most of them have some bootstrap option or the other :)
lennert> kedars: yeah, which have rotten and don't work :)
lennert> kedars: e.g. ant is a lot of fun
lennert> kedars: it needs 3 or 4 passes or so
kedars> oh!
kedars> rpmbuild -ba --define 'with_bootstrap 1'  /root/rpmbuild/SPECS/ant.spec
kedars> error: Failed build dependencies:
kedars> :)
lennert> don't think it'll be easy! :)
sharkcz> we were bootstrapping the java stuff on s390x recently and I can tell you it's a nightmare :-)
* kedars shudders
sharkcz> everything is in https://s390.koji.fedoraproject.org/koji/builds so you can use it as a reference

Sounds like fun ! :-)

Fedora-ARM 11 available!

Fedora-11 ARM Repositories are now available!

Details: built for ARMv5 EABI, soft-float, little endian.

Currently there are packages from three yum groups: Base, Core and Web-Server.

A yum-enabled pre-built root filesystem is available along with instructions for working with known platforms (currently Qemu,
Sheevaplug)

The Fedora/ARM architecture wiki page has more information:
http://fedoraproject.org/wiki/Architectures/ARM

Do let us know any issues that you may face, or any additional Fedora Packages/Groups that you would like to have in the
repositories.

Our next steps include a) making the package repository much more complete by building packages from more package groups (starting
with the Gnome Desktop group) b) tracking f11-updates for these packages.

Incorrect host parameter

While rebuilding gcc and glibc I found out that the builds were being passed the host parameter as

--host=armv5tel-koji-linux-gnu

instead of

--host=armv5tel-koji-linux-gnueabi

Strange!

I checked back at the gcc/glibc that were built earlier (27 April 2009). These seemed to get built correctly with the right host parameter. So what changed? The rpm and redhat-rpm-config rpms were the same as in April. And yet we see a different behaviour.

Normally, the host parameter that we see while manually building is armv5tel-redhat-linux. While right now we see it is armv5tel-koji-linux. So most likely koji is substituting the _host rpm macro with its own version, and while doing so it neglects the -gnueabi suffix. And probably it was included in one of the latest versions (We use koji trunk on fedora-arm)

Thankfully, there is a configuration option to override this through kojid.conf. Fixed it on all the builders.

Minimal fedora-arm rootfs

The bootstrap phase has seen a lot of packages being built for F-11. Specifically, 730+ packages have been built (and counting).

So how far have we reached as far as a working rfs is concerned? Not too far away now. Only some dependencies need to satisfied. All the major dep trees have been satisfied. The only thing remaining is cyrus-sasl which needs mysql and postgresql. Once that is done, we'll give another shot at building an rfs from the available packages. If no new install-time deps are required, we'll be good to go!

Any time now!

Autobuild result

We built 117 packages in the autobuild run that we ran through the weekend. Much more than what I was expecting, great! We missed out to tag the 'libtool' package in the tag, so nothing that uses libtool was built. That will happen in the next run tonight.

Second phase of the bootstrap build ..

After some hiccups, the second pass for the F-11 builds from F11` has been finally scheduled.

Yesterday, there was an issue with unicode encoding in some package's "provides" tag.
Basically, there is a koji utility called "mergerepos" which (as the name suggests) merges multiple repos into one. This is required because we use something called "external repos" (The F-11` repo is attached as an external repo to the build tag). Since, we need to merge the repo generated out of the build tag and the external repo, mergerepos comes into the picture. Now, some of the font packages (baekmuk-ttf-batang-fonts-2.2-21.fc11.noarch.rpm, baekmuk-ttf-gulim-fonts-2.2-21.fc11.noarch.rpm, baekmuk-ttf-dotum-fonts-2.2-21.fc11.noarch.rpm, baekmuk-ttf-hline-fonts-2.2-21.fc11.noarch.rpm, sazanami-mincho-fonts-0.20040629-7.20061016.fc11.noarch.rpm) had unicode/random encoding in their "provides" tag. "mergerepos" didn't seem to handle it pretty well and it used to fail.
Since, luckily, not many packages seem to depend on these font packages, we have removed this packages from the F-11` set.
(We have reported the issue to the maintainer and he is having a look at it)

After this, some time was spent on coming to a closure. There were some missing deps here and there. Nothing major here. Just had to add these packages to the dist-f11-build-bs tag and we were good to go.

The second pass has started now. Hoping that a good set will be built over the weekend :)

Peek Forward to autobuild

A fairly large set of F-11` rpms (from the coreset) is now available. There are about 40 odd packages which have still to make there way into this thing yet.

But in order to make automation work over the weekend, we'll go ahead and schedule a build pass over all the 700SRPMS using the F-11` build root. There will still be failures (since those 40 packages haven't made there way into F-11` yet.) And we will reschedule them after the failing-40 are done. But there should be some good number of packages that succeed. And it validates the build scripts as well.

Lets see how this goes.

Circular Deps

The most time-consuming component of tracking Fedora package builds is circular dependencies... More the number of packages in the set, more the time it takes. Yesterday was spent in chasing a libsoup->libproxy->webkitgtk toolkit dependency.

This is how it starts:

Build --> gvfs-1.2.3-2

Dep failure. libsoup. Ok.

Build --> libsoup-2_26_0_9-1_fc11

Dep failure. libproxy. No problem.

Build --> libproxy-0_2_3-8_fc11

Error WebKit-Gtk-devel not found. Hmm... Strange I see it right there in koji, with the proper tag, why doesn't it find it?

Turns out the package was renamed to webkitgtk in F-11. So it won't pick the older package. Alright, lets build that.

Build --> webkitgtk-1_1_4-1_fc11

Ok, webkitgtk again needs some version of libsoup that I don't have in koji yet.

Well, but libsoup needs libproxy again. Going through libsoup commit logs, tells me that libsoup shifted from GConf2 to libproxy in the past. So lets build libsoup's version that builds with GConf2.

Build --> libsoup-2_25_4-1_fc11

No, that would have been too simple won't it. The version of GConf2 that I have in koji has some 'pkgconfig' errors. So I look at GConf2 commit logs. Somewhere upthere, there is a version that fixes this issue.

Build --> GConf2-2_24_0-3_fc11

Finally! Something got built... And now to pop the rest of the stack and pray :-)

Its time I start looking into how to fix the problem with shadowing.

Slow slow arm koji

For the past couple of days, arm koji has been horribly slow... Opening the front page or task status page is like a 1 minute task. Or it dumps a "proxy error"

genpkgmetadata is probably killing the VM.

Quick rehash

A quick rehash of what all has been happening thus far:

F-11 tracking is in-progress. koji-shadow was shadowing builds for f-11 for the main fedora koji server. I was expecting it to run more independently, but that hasn't yet happened because of the "prefer-new" option. It pulls in additional dependencies, that have to be taken care of manually.

Recently, koji-shadow is also running into some Python Traceback Issues. I have requested dgilmore to look into it.

In the meanwhile, we are using koji to perform an F-11 bootstrap the way we did for F-10. The only difference being, this time it will be done using koji. If we get there with this method faster than koji-shadow, great!

F-10 --> F-11` --> F-11 (for the basic set of 2000 rpms or ~450 SRPMS)

Thanks to koji and scripting, we have already built 400 of these SRPMS for the F-11` within a span of 3 days. The others are build/dep failures.

But what that means is that, once we have F-11' complete (with patched packages etc.) going to F-11 will only be a 5/6 day effort, since build deps are already fixed.

Now to fix the deps failures first...

The Fedora-ARM Blog

First blog entry for the Fedora-ARM blog.

A collections of blogs from all contributors to Fedora-ARM.