systemcall dot org

Every now and then, when I’m bored with normal work, I tend to my personal projects. For some reason, that tends to be mid-year, every year. So, here we go…

Old ideas, new style

This time was an old time project, to break the parallelization barrier without the use of locks. I knew simple boolean logic on regular Turing machines wouldn’t do, as they rely on the fact that the tape is only written by one head at a time. If you have more than one head writing to the same place, the behaviour is undefined. I’ve looked into several other technologies…

Some were just variations of Turing machines, like molecular, ternary and atomtronic computers. Others were simply interesting toys, like the programmable water, but there were two promising classes: quantum computers (taking a bit too long to impress) and non-deterministic Turing machine (NTM), that could get around some concurrency problems quite well.

Quantum computers are nice, but they’re more like vector computing on steroids. You can operate on several qbits at once, maybe even use non-locality to speed up communications, but until all that reaches the size of a small building, we need something different than yet-another n-core frying-pan Intel design. That may be non-deterministic machines…

Non-deterministic Turing machines are exactly like their deterministic counterparts, but the decision process can take different paths given the same input. So, instead of

if (a == 10) do_something;

you have

if (probability(a) > threshold) do_something;.

The problem with that is that you still can’t ignore different processes writing to the same data, since if a process happens to read it (even if by chance), the data you get from it is still being concurrently changed, therefore, still undefined. In other words, even if the state change is probabilistic, the data is still deterministic.

So, how can you make sure that, no matter how many processes are writing to a piece of data at random times, you still have meaningful data?

Well, if the reading part is not interested into one particular value of that data, but say, only in the probability of that data be a certain value, then it doesn’t matter in which order the writes are performed, as long as they’re all writing about the same thing. That can be viewed as a non-deterministic register, not a non-deterministic Turing machine. You can still have a deterministic machine reading and writing on non-deterministic registers, that you still won’t need locks for those values.

Probabilistic registers

Suppose you want to calculate an integral. You can use several very good deterministic algorithms (like the Romberg’s method), but if you create producers to get the values and consumers to reduce it to a sum, you’ll need a synchronized queue (ie. locks) to make sure the consumer is getting ALL the values. If you loose some values, you might get a completely wrong result.

If instead, you create random generators, getting the value of any (multi-dimensional) function from random values, and accumulate them into a non-deterministic register (say by updating the average of the last values, like recharging a capacitor that keeps discharging), when the consumer process read the value from it, it’ll get an average of the previously written values. Because the writes are all random, and if you keep only the average of the last N writes, you know how much data is being filled in and what’s the average value for that block.

In essence, you’re not storing any particular value, but information enough to re-create the original data. Since specific values are not important in this case, the order in what they appear (or if they appear at all) is completely irrelevant.

In a way, it’s similar to quantum mechanics, where you have the distribution of probabilities but not the values themselves. With a few tricks you can extract every information from those distributions. This is also similar to what Monte Carlo methods are. Taking the probabilistic side of computations in order to achieve a rough image of the problem quicker than by brute-force, when the complexity of the problem would be non-polynomial and the problem size would be big enough to need a few billion years to calculate.

Here are some examples of a non-det register.

A new logic

Obviously, current deterministic algorithms cannot run on such machine. All of them are based on the fact that specific data is stored in a specific place. If you compress a file and uncompress it back again, it’s just impossible to work with probable values, your original file will be completely undecipherable. Also, boolean logic cannot operate on such registers, Operating an AND between probabilities or a XOR of two averages doesn’t make any sense. You need a different logic, something that can make sense of probable inputs and analog streams. Fortunately, Bayesian logic is just the thing for such machines.

But the success of Bayesian logic in computing is not enormous. First, it’s not simple to create algorithms using a network of probabilities, and very few mundane algorithms can benefit from it to be worth the trouble. (Optimor labs is a good example). But with the advent of this new machine, a different logic has to be applied and new algorithms must be developed to make use of it.

Probably by mixing deterministic with non-deterministic hardware and creating a few instructions that would use the few non-deterministic registers (or accumulators) available. Most likely they would be used for inter-process communication in the beginning, but as you increase the number of those registers you can start building complex Bayesian networks or also neural networks in the registers themselves. That would open the door to much more complex algorithms, probably most of them unpredictable or found by trial and error (genetic approach?), but with time, people would begin to think that way and that would be the dawn of the non-deterministic computing.

Eureka moment

The probabilistic register was an Eureka moment, when I finally implemented it and saw that my simple monte-carlo or my neuron model was really a three-line code, but that didn’t last long…

A few days ago I read the news that DARPA has developed a fully featured probabilistic processor, taking random computation to the masses, even using Bayesian logic in place of boolean logic! The obvious place for a “public trial” was spam filtering (though I think the US military is not that interested into filtering spam…), but according to them, the sky is the limit. I do have to agree…

Anyway, I wouldn’t have 5 spare years nor $18mi in the bank to implement that project like they did, so in a way I’m glad that it turned out to be a good idea after all. I just hope that they make good use of it (ie. not use it for Carnivore 2.0), and they realize that not only the military have uses for it, or anti-spam filters, for that matter. We (them?) should develop a whole new set of algorithms to work on that logic and make good use of a hardware that is not a simple progression of current technology (to keep Moore’s law going), but could be a game changing in the next decades.

I could re-focus on defining the basic blocks for that processor’s new logic, but I’m sure DARPA has much better personnel to do that. So, what’s next, then? I’ll tell you next year…

Disclaimer: This is not a rant

I spent my last year working with Java, and it was not at all bad. But while Java has its moments and shines, I always felt a bit out of place when using it. In fact, when I moved back to C++, contrary to when I moved to Java, I felt that I actually wasn’t missing much…

Last year, while writing in Java at work, I felt compelled more often than usual to write C++ programs at home. Even simple programs, that would do better with scripting languages, they all came in C++.

Recently, working full time with C++, I noticed I’m doing very little home development and definitely not doing any Java. So, what did I miss about C++ that I don’t miss about Java?

Expressiveness: While functional languages are much more expressive than C++, there are few languages less expressive than Java. Java encourages child-like programming like forcing to call everything by methods not operators. By not having explicit pointers, operator overload and other dangerous things from C++, you end up repeating yourself quite a lot and it’s very hard to understand the logic afterwards, when all you have is bloatware.

While Java designers tried to avoid pointers and operators, they couldn’t. We still have null references (throwing null pointer exceptions) and the fake operators (like toString(), hash(), compare()) that can easily be overridden to change the expected behaviour pretty much the same way as C++ operators, but in the “method” notation.

In the end, you can do some bad things, but not all. So, they took away dangers by taking away functionality, without a proper redesign of C++.

Abuse of Object Orientation: While in Ruby, everything is an object, in Java, almost everything can be. Every class derive from Object silently, but base types do not. So you have the basic objects (Integer et al) which get automatically converted into basic types in subtle ways it’s hard to predict and has a huge performance impact (see auto-boxing).

Not just performance, but the language design is, again, incomplete.

Most OO programmers (mainly Java ones) complain a lot about Perl OO. They say Perl (or Python for that matter) has no proper OO, since everything is a hash and there is no concept of protection.

While Java objects and members are strongly typed, and you have the concept of protection, it’s way too easy to transform Java OO into Perl OO with reflection.

Of course, with C++ you can cast things to void pointers, mess up in the memory and so on, but getting objects by name, removing the private protection in a safe way is simply wrong. It’s like giving loaded guns to children and telling them where the lock is.

Abuse of Design Patterns: Java developers are encourage to use design patterns, to the point of stupidity. The first thing I learnt from design patterns is that their misuse is actually an anti-pattern.

Properties are important when the requirements change too often, not when they’re static. Factories are used when the objects created may differ or be customized, not for never-changing one-object construction. Still, most libraries (all?) will have Factories, Properties and so on, just for the sake of Design Patterns Compliance ™.

Fact, one of the strengths of Java development is that every one is encouraged to do things the same way. No Larry Wall style, all factory workers, doing their share in the big picture. While this is good for big, quick projects on companies with high turn-over (like consultancy companies), it’s horrible for start-ups or more creative development.

Half-implemented features: Well, templates is an issue. There is no template mechanism in Java. With the so-called Generics (like cheap version of meds), there is no type safety at all, it’s just syntax sugar for lists of Objects.

That generates a lot of misunderstandings and bad code being generated when the syntax is obviously correct, that is, if the types were actually being checked.

Again, incomplete design for the sake of backward compatibility with old codes and VMs.

Performance: Running in a JVM is already a bad start for performance, but a good compiler and a well done JIT environment can take most of it away by intelligently removing unused code, re-optimizing most used code during run-time and using profiling results to change branch-prediction code.

While the JVM does some of it, it also introduces several problems that take away the advantage and put it back on the back of the class. Auto-boxing and generics create a lot of useless casts, that can be a huge performance hit. Very few Java programmers really care about it and the compiler doesn’t do a good job in reducing that impact or even warning the programmer.

I often see Java developer scorn at performance issues. The phrase used most is “a programmer shouldn’t care about memory footprint or performance, only about business logic”. That, together with the fact that almost all universities now are teaching Java in undergraduate courses, kinda frightens me a bit.

Strong dependency on IDEs: Borland made quite a lot of money out of C++ IDEs in the 90′s, but most C++ programmers I know still use VIM or Emacs. On the other hand, every Java programmer I know use Eclipse, IntelliJ or something of the sort.

This is not just ease of use (code completion, syntax colouring, hints, navigation), it’s all about speeding up the development process by taking away boiler-plate code generation and refactoring.

IDEs are capable of writing complete pieces of code, refactor and re-write things (even behind your back). The programmers don’t care about it, the code becomes bloated, unintelligible and forgotten. Not to mention the desire of IDEs and people following IDE-style to use certain patterns for everything, like using Properties where simple structures would suffice. (see above, Abuse of Design-Patterns).

False Guarantees: The big selling point of Java, besides cheap cross-platform development, is it’s apparent safety and ease of use. But it isn’t in so many levels…

The abuses and problems related above are only part of the story. The garbage collector is another…

Some good garbage collection routines can help the initial development of programs, and they do take away the job of the lazy programmers to manage their own memory, but the Java garbage collection became a beast, with incomprehensible command-line options, undefined behaviour and total lack of control over it. You’re rendered hostage to its desires.

Not to mention the complete memory management that won’t cope with dynamic memory allocation. I mean, if you want to make memory management easy for programmers (as they went to all that trouble for a garbage collection), you could have gone a bit further and actually figured out the available memory and used it politely.

Join those with the fact that pointers and operators are still available, and you have a language that is not so much simpler than C++, with a huge price in performance and weirdness.

Undocumented APIs: Java claims to be platform independent, but has quite a few available (but undocumented) APIs to use platforms specific functionality (like signals). Still, Sun (now Oracle) reserves the right to change whenever they wish and there’s little you (or anyone) can do about it.

And that takes us to the final point:

Standards (or lack thereof): Sun did a nice job at many things (mostly hardware and OS), but they screwed up neatly when it came to support software. There is no standard, IBM and even Microsoft created their own JVM (which was better than Sun’s, btw) without any final definition about the standard API. During the Java 1.1 days, it was possible to be platform agnostic but VM specific in the same platform!

Conclusion: Java was meant to be an easy language, but it turns out that it’s deceitful enough to be just as bad as any other. And recent changes are making it worse.

Programmers are loosing the ability to understand how the machine works, how their languages behave and, more importantly, to know the implications of their actions.

Why spend time understanding the fiddlings some people had with Java if you can spend the same time understanding how the machines actually work and therefore be able to use any programming language you want?

Some argue that Java is the new Cobol and will disappear the same way… I tend to agree…

Gorik was a methodical prisoner. Every morning, precisely at 6 o’clock, he’d wake up, wash his face and prepare for the morning food rations. After eating and tidying his bed, he’d go outside for the sun bath.

His schedule wasn’t particularly full, but he had some tasks he liked to do. Washing the court, preparing dinner, helping people in the library and most of all, teaching mathematics to the children. He was not a great mathematician, but among the prisoners, he was the most gifted.

Although his race was known to be violent, and many others tried to escape prison, he was a peaceful man. Not once he tried or even hinted to escape, nor he participated in the few mass escapes that happened since they’re all imprisoned. As with most prisoners of war, they weren’t criminals or had done anything more terrible than those that had imprisoned them, they just were unlucky enough to be on the loosing side of a major war.

Violent or not, their race was proud of what they achieved and were no more destructive than their enemies, nor they actually started the fight. But that doesn’t concern us now, the important matter now is that Gorik was not in his cell at 6 o’clock this morning to get his rations.

The janitor’s face had but one tear rolling down his cheek. The key in his hand opening a door that didn’t need opening. Pointless exercise but he continued, by force of habit, if nothing else. He was definitely not there.

Others were coming to see, but the sense of nothing was global. There wasn’t a single man, on either side, with his mouth closed. Some were dripping, what could be more tears, or saliva. It didn’t matter any more. He was not there at all.

No alarm was sound, no one ran. There was nothing to do. There were no broken doors, no knocked-down guards, no bribes paid (or no one said so), nor any camera caught anything convincing. He vanished.

Life continued in the prison. There was no one to help with the dinner today, or anyone to help the children in their assignments. It was not the same without him. Where did he go? And why now? There must have been something really serious to take him that sudden, and silently. He just left.

Two days later, when the janitor had the courage to enter his cell to clean it up, he noticed a letter on the bed. Nothing special, just a letter half-inserted back into the light brown envelope. The writing on the envelope was a bit wobbly, like children writing. It was addressed to him personally, and being only a letter, there was no point in not delivering it to him.

He was not there any more, and he left the letter in plain sight. That unofficially gives one the right to read it, I guess. Well, the janitor agreed, and opened the letter. It was from Gorik’s wife.

She was also a prisoner, but ever since the war was over, women and children got moved to a more decent accommodation, where the children could learn how much better was life on the other side and interact with the children on this side, and forget about their horrible and violent past.

The note said: “Dear Gorik, I’m not feeling well lately and Juma is a fine woman already. She’s left with a good boy and I’m afraid I’ll be alone for the rest of my days. I sincerely hope you are in better company than I am. Love, your wife.”

The next day, the prison manager went personally to the apartments she was living, and asking around learnt that nobody saw Gorik since the end of the war, but his wife had not showed up for a few days, also.

He took the note from his pocket, in which his secretary had written the block and apartment number. After a few minutes walking in circles, he managed to find the block and the correct door, which was half-open.

Knocked once. No answer. Twice. Nothing. The manager peered into the room by the two centimetres available, but all he could see was that the TV was on. Feeling a bit guilty for doing so, even for a prison manager, he opened the door a bit and found no one in the living room. A few steps inside, a door to the left. Empty bathroom. Another door to the right, empty room. Down the small corridor, there was a door, completely open and letting the sun, that was shining on the other side of the building, come through with all its might.

Barely visible among the flood of light, a pair of feet. No, actually, two pairs. Curiosity was not in the manager’s list of sins, but he could no longer wait. Sweating and his heart pumping, he crossed the room, just to find two people lying down on the bed. Calm as summer night.

Nothing to do with the topic of this blog, you are right, but I couldn’t stop thinking about the reality England faces regarding football.

All these years watching a mediocre football, even in the Premier League, where Cristiano Ronaldo is the top player and where Robinho can’t play football made me wondering what’s terribly wrong with the nation’s love for the sport. Not just passion, but the English people were the ones that invented the sport. Here in Cambridge its very first rules were written and the first match with those rules played in Parker’s Piece in 1848.

With more than 160 years of football history you would imagine that they’d have a bit more skills… Recently I have found a perfectly good reason why this happens.

Another English passion, maybe even more important that football, is lawn. Not cricket, not complaining, not political jokes: lawn. Mediocre football is all right, mediocre grass is a crime. But, grass and football are very much connected, probably the very reason why they loved to play it in the past, but as priorities are laid, lawn apparently comes first. In my recent visit to my son’s new school, with very impressive lawns, bigger than an official football pitch. The “football pitch”? It’s made of asphalt, which is also used as car park some times…

Baffled as I was, when I asked the children about football, they said they were allowed to play, but they had to bring their own balls. So far so good, but then it came reality: there is a recommendation to use foam or rubber balls, to avoid injury. Question is, what does more injury, proper footballs or asphalt?

In Brazil, children as young as 1 year old play football with anything. Coconuts, food cans, socks rolled into a ball, even stones. And they usually play barefoot, on dirt, or even asphalt. Health and Safety is important, but not to the point of removing completely the fun of being a child. If a child doesn’t get dirty, it won’t learn to clean itself, to avoid the situation in the future and, more importantly, to understand the pain of living and how good that feels.

European football? Never heard…

It’s been five years now that I switched from Debian to Ubuntu, primarily for the updated software and radical changes in the user interface, and there are quite a few things that were constant all this time. When on Debian, I always used the unstable branch. It was the obvious choice for a non-mission-critical desktop environment I always needed. But even being unstable, it lacked a bit of risk-taking that made me some times having to compile (or download binary) applications by myself, working around the packaging management system.

With Ubuntu, it’s the exact opposite. The ongoing lack of support for nVidia and ATI boards, PulseAudio and the new Plymouth splash are good examples of major failures on deploying a technology that is yet too young to be in a distribution, especially a Long-Term-Support one. Recent rumours on changing Firefox to Chrome is a more critical change, since the whole community around Firefox (add-ons, plug-ins, bookmarklets, etc) cannot easily be migrated to Chrome or any other major browser. But this is all about the Ubuntu Way.

Identity

Ubuntu, like many other Linux distributions (especially Debian), has built its identity around the OS that most users share. It’s organic, and grows with time and feedback from the users, joined with the directions the “board” is taking in what goes in and what goes out. The original Linux community (back in mid-90′s) was a bit homogeneous in that respect, with most distributions being yet-another-collections-of-packages, be it RPM, DEB, Tar balls or anything else. With time, strong feelings were separating some distributions apart, and specializing others. Debian, for instance, became over preoccupied with license issues (no other than open source was allowed), while RedHat became more enterprise focused, flooded with third-party libraries, commercial products and a licensing scheme that was more like Microsoft than anything else.

Still, within the Debian community, some people (like me) thought that the release schedule was too wide and the licensing issues were too narrow to produce a really helpful desktop replacement for other commercial systems, like MacOS. Indeed, after a few releases, Ubuntu has shown that it can replace them for most uses to most users. I, as a Linux user for so many years, welcomed the ease of use of a MacOS without the lock-downs and lame packaging systems.

But they went further, and decided to be very (very) much the same as Apple. Initially, the Linux way was to offer everything there was available for everything. There were dozens of instant messengers, browsers, picture viewers, consoles, etc, all installed by default (or to pick from a selection of thousands of packages in the installation process), which was a major pain. Recently, Ubuntu has provided an installation process easier than Windows and MacOS, and for every application type, there was only one default option. That is, what has become, the Ubuntu Identity.

Taking Risks

To keep that identity, and still progress as fast as they (and me) would like, one has to take risks. I have to say that, for the most part, they were right on the spot. Some failures (as mentioned) are expected to happen and you are left with the consequences and decision of those risks. For a company with such a tight budget (and such high expectancy), there is little they can do differently. If they had bigger budgets, they could spend more time adapting the proprietary graphic drivers and the update system (that never works on fine-tuned machines), but they don’t. And based on how updates work on Windows and MacOS (ie. they don’t), I’m not surprised with Canonical’s failures.

I like Firefox, ALSA and Pidgin, but if the overall experience is more stable (and complete) with Empathy, Chrome and PulseAudio, so be it. We’re passt the time to complain about personal preferences in favour of a wider viewpoint. I’m too old to rant about how pity is the new splash screen when using ATI proprietary drivers for the time being, I just want to install and run. As long as my VIM is working and there is a browser and an IM to use, I’m happy. I don’t care Gimp is not included by default, I do dislike that GCC is not, but I understand the reasons and always install it first thing when I get a new system.

That’s the Ubuntu identity and the risks Canonical takes to move the desktop experience forward. As unstable Debian people used to say, that’s the risk of being on the edge…

Upgrades never work

So, I stated that upgrades never work for fine-tuned machines, and that has been my experiences until today. In the beginning, I thought it was that Ubuntu was still immature, but today I had to roll-back my Lucid installation I did yesterday for major incompatibility issues, mainly with the ATI proprietary graphic driver (splash and return from sleep).

So far, the only way I can upgrade Ubuntu is by installing a complete new copy of it every time, and apply the backed-up changes in configuration files manually after all is done. It may seem a lot of work, but every time I try to upgrade and every time I end up installing from scratch and applying the few manual tuning later. Now that I know exactly what I have to change and where (after years of doing), it takes me roughly 15 minutes to customize it.

My configuration is in such a state that it takes me zero maintenance and little backup disk space, as well as easy installation process. The magic is simple.

Preparation

This is one thing I recommend to any system, Linux, Windows and MacOS: Split into, at least, two partitions. One, around 50-80 GB, for your system, preferably the first one (primary partition). The other(s), taking up the rest of the hard-drive, for your data/home directories. If using Linux, of course, reserve (at the end), a space for your swap (4GB is more than enough, even if you have that, or more, of RAM). Swap is a safety measure and not to be used under any normal circumstance.

Daily Usage

Backup your home directory often, including personal configuration files, IM history, panel short-cuts, everything. Apart from your data, the rest might give rise to some complications when upgrading the user environment (Gnome, KDE) but that’s minor and can be overcame easily. That will help you in case things go awry in your update/replace process. A cron job or manual invocation to a script is recommended for that.

Also, remember to back up (manually, by copying) every system configuration you change. Because most configuration on Linux is a text file, that part is very easy. It has to be done manually because, as it’s very simple and easy (you shouldn’t change that many configuration files), you can do a detailed comparison between what’s in there and what you want to replace or add. This will be important for your post-update process.

Additionally, any non-essential data can be moved to a shared disk (with appropriated backup), accessible over the network. This way, you not only don’t have to backup all your data (photos, videos, documents) every install (could take days), but they will also be available from other computers while you upgrade your machine, so you can continue working on them as soon as your machine is ready.

Upgrade

Upgrades never work, especially if you have changed the configuration. Some systems evolve and can’t read old configurations properly, new systems won’t read other systems configurations and migration scripts never work properly on modified files. What’s worse, as Ubuntu has its own identity, the new systems will work better (or only work) with other new systems. So the integration between the new systems and your old, changed, systems will most likely fail silently. PulseAudio is the best example of that conflict.

To update, simply re-install the new version from CD (USB, or whatever) into the OS partition. So far, they have managed to make the upgrade to new systems pretty easy, if you discard your old ones. Empathy imports pidgin accounts (and history), all basic systems are properly configured if you do a fresh install. As wireless network passwords, panels, personal short-cuts, and other configurations are stored in your home directory, you just have to log in to see your old desktop, just the way it was.

The few things that aren’t installed (like GCC, VIM, gstreamer plugins) can be easily installed if you have a list of things you always install in a file (in your home dir), like build-essentials, ubuntu-restricted-extras bundles. Printer VPN, printer and share configurations can be easily copied over from your backup as soon as you installed and an apt-get upgrade can be done to get the new stuff since the CD was released.

Roll back

What’s best in this strategy is that roll backs are extremely easy. You can’t roll back a dist-upgrade using apt, but you can safely re-install the previous CD in case it breaks up things so badly it becomes unusable. Like the new Ubuntu, it’s still bad with proprietary graphic drivers and the open source ones are not nearly as good. So I just rolled back and will wait until it stabilises.

Instabilities occur most often in Long-Term-Support releases (like the current). It might seem weird, but it’s pretty simple: they commit to three to five years of support, so they must get new software that will last that time. The lifetime of open source projects is not great (still, longer than many commercial products), but a five year commitment on a software that is already five years old is a big risk. Ext3 and Ext4 filesystems are a good example of this case.

So, instead of providing the stable components, they change radically the interface and sub-systems and wait for them to stabilise, hoping that the production state of the release will remind developers to speed up the fixes. While not optimal to the users, it’s more or less the only way they can go without breaking the promise of support when the application goes dead. This is why enterprise Linux is so expensive, because companies require stability as well as support, and ultimately, the distribution companies will have to maintain some of the dead application for years, if not decades.

Not only roll back is easy, but changing distribution entirely. As your data is distribution agnostic (Linux centric, not package-system centric), you can re-install virtually any other Linux distribution, as many times as you want, and keep the same look and feel.

Conclusion

In summary, it might look more complicated to use and maintain, but it’s not. Once your setup is done (partitions, backup scripts), the rest is pretty easy and quick. So far, I have stubbornly upgraded every release (since 7.04) to make sure it’s still harder than re-installing and it has been the case for every release.

Also, if you have nVidia or ATI graphic boards, never upgrade in less than a month after the release is out. I recommend you upgrade at least two or three months later (mid-releases), as most of the vendors will have updated to match the new Ubuntu Way.

Lastly, as I normally fine-tune my computer, I haven’t had a successful migration of any operating system until today. I always try to upgrade, if available, and end up re-installing everything. That was true with DOS, Linux and Windows, since 1990 and I doubt it’ll change any time soon. It’ll be necessary an intelligent installation process (which our computers are not able to run, yet), to do that.

In the far future, it lies, then.

I’m not the one to normally do reviews or ads, but this is one well worth doing. Humble bundle is an initiative hosted by Wolfire studio, in which five other studios (2D Boy, Bit Blot, Cryptic Sea, Frictional Games and the recently joined Amanita Design) joined their award-winning indie games into a bundle with two charities (EFF and Child’s Play) that you can pay whatever you want, to be shared amongst them.

All games work on Linux and Mac (as well as Windows), are of excellent quality (I loved them) and separately would cost around 80 bucks. The average buy price for the bundle is around $8.50, but some people have paid $1000 already. Funny, though, that now they’re separating the average per platform, and Linux users pay, on average, $14 while Windows users pay $7, with Mac in between. A clear message to professional game studios out there, isn’t it?

About the games, they’re the type that are always fun to play and don’t try to be more than they should. There are no state-of-the-art 3D graphics, blood, bullets and zillions of details, but they’re solid, consistent and plain fun. I already had World of Goo (from 2D Boy) and loved it. All the rest I discovered with the bundle and I have to say that I was not expecting them to be that good. The only bad news is that you have only one more day to buy them, so hurry, get your bundle now while it’s still available.

The games

World of Goo: Maybe the most famous of all, it’s even available for Wii. It’s addictive and family friendly, has many tricks and very clever levels to play. It’s a very simple concept, balls stick to other balls and you have to reach the pipe to save them. But what they’ve done with that simple concept was a powerful and very clever combination of physical properties that give the game an extra challenge. What most impressed me was the way physics was embedded in the game. Things have weight and momentum, sticks break if the momentum is too great, some balls weight less than air and float, while others burn in contact with fire. A masterpiece.

Aquaria: I thought this would be the least interesting of all, but I was wrong. Very wrong. The graphics and music are very nice and the physics of the game is well built, but the way the game builds up is the best. It’s a mix of Ecco with Loom, where you’re a sea creature (mermaid?) and have to sing songs to get powers or to interact with the game. The more you play, the more you discover new things and the more powerful you become. Really clever and a bit more addictive than I was waiting for…

Gish: You are a tar ball (not the Unix tar, though) and have to go through tunnels with dangers to find your tar girl (?). The story is stupid, but the game is fun. You can be slippery or sticky to interact with the maze and some elements that have simple physics, which add some fun. There are also some enemies to make it more difficult. Sometimes it’s a bit annoying, when it depends more on luck (if you get the timing of many things right in a row) than actually logic or skill. The save style is also not the best, I was on the fourth level and asked for a reset (to restart the fourth level again), but it reset the whole thing and sent me to the first level, which I’m not playing again. The music is great, though.

Lugaru HD: A 3D Lara Croft bloody kung-fu bunny style. The background story is more for necessity of having one than actually relevant. The idea is to go on skirmishing, cutting jugulars, sneaking and knocking down characters in the game as you go along. The 3D graphics are not particularly impressive and the camera is not innovative, but the game has some charm for those that like a fight for the sake of fights. Funny.

Penumbra: If you like being scared, this is your game. It’s rated 16+ and you can see very little while playing. But you can hear things growling, your own heart beating and the best part is when you see something that scares the hell out of you and you despair and give away your hide out. The graphics are good, simple but well cared for. The effects (blurs, fades, night vision, fear) are very well done and in sync with the game and story. The interface is pretty simple and impressively easy, making the game much more fun than the traditional FPS I’ve played so far. The best part is, you don’t fight, you hide and run. It remembers me Thief, where fighting is the last thing you want to do, but with the difference is that in Thief, you could, in this one, you’re a puss. If you fight, you’ll most likely die.

Samorost 2: It’s a flash game, that’s all I know. Flash is not particularly stable on any platform and Linux is especially unstable, so I couldn’t make it run in the first attempt. For me, and most gamers I know, a game has to work. This is why it’s so hard to play early open source games, because you’re looking for a few minutes of fun and not actually fiddling with your system. I have spent more time writing this paragraph than trying to play Samorost and I will only try it again if I upgrade my Linux (in hoping the Flash problem will go away by itself). Pity.

Well, that’s it. Go and get your humble bundle that it’s well worth, plus you help some other people in the process. Helping indie studios is very important for me. First, it levels the play-field and help them grow. Second, they tend to be much more platform independent, and decent games for Linux are scarce. Last, they tend to have the best ideas. Most game studios license one or two game engines and create dozens of similar games with that, in hope to get more value for their money. Also, they tend to stick with the current ideas that sell, instead of innovating.

By buying the bundle you are, at the very least, helping to have better games in the future.

First time we went back to Brazil, besides the 40 degrees heat at nights and family discussions about how much more time we spent at the other aunt’s, it was no big deal. So the idea the second time we not to plan better and not to spoil the family any more. That time, there was little discussion, but we forgot to check for the little one’s passport and spent two weeks on queues at the federal police to get it, involving sending his documents over from UK via Fedex (thanks Guto!).

A lot of time (and planning) later, we decided to go, our style, and check all documents before flying. It was wonderful! Besides the usual family fight, all went well. We went to the beach and it was sunny and fantastic. On our way back from the beach I said: “We did it! Nothing went wrong this time, eh?”. It came out as a relief, but turned out to be a curse! At the exact moment I said that, a giant pool of lava decided to come out into existence, as if feeling my moment of joy, just to spoil it. It was a masterpiece, from a Norse god or something.

Wednesday, 14th April 2010

All flights stopped, and on Wednesday itself I knew ours (on Sunday) would be included. As business continuity planning (BCP), I coordinated with my company, my family and KLM to get the most out of the zillions of different news flashing on Twitter, Facebook and BBC.

I have to say that we were a bit disappointed with Eurocontrol. They stopped all flights for safety reason (quite reasonable) but they sat and waited until the volcano would stop spilling lava. As if that would solve anything, even after all geologists said it could take months, or even years, to stop.

Friday, 16th April 2010

Feeling the pressure of $200mi a day each, all airlines got their planes flying with no passengers. Some flew at lower altitudes, most at regular and some even got right into the ash, just for fun (and results). And even with all results negative, Eurocontrol delayed days to slowly restart flights.

Monday, 19th April 2010

We were still stranded in Brazil, with two hyper-active boys, mum without her medicines and dad without his nerves in the right place. We were offered a flight back, but KLM warned it was only to Amsterdam, so we didn’t take it. I said to the advisor to only book us if the flight was all the way to London.

Tuesday, 19th April 2010

We were booked on the same flight as before, but, according to KLM, all the way to London. I was amazed. KLM had done some flights to London already and I thought we were going home, much easier than I had foreseen… how naive.

At the Guarulhos airport, KLM said they couldn’t guarantee both flights, which came as a surprise. But with all the chaos, I thought it was only fair to accept their apologies and proceed with our travel back.

Wednesday, 20th April 2010

But it was only when we got to Schipol that I really understood what chaos really meant. There were a few thousands of passengers queueing for no apparent reason, going in random directions, being directed by random people in random uniforms, working for random companies, pointing to nowhere in particular.

One guy, dressed with a KLM suit, said I should get information directly from KLM (I didn’t ask). Another said that, in order to get info, I should take the queue that would take approximately 2 hours. When I finally got to the electronic ticket machine and had difficulties, the helping person told me she could not help because the machine was not helping.

In the end, we found out that our flight to London (KL1017) never existed. There were other flights to London (KL1019), but they were all full, and there were no confirmed flights in the next day. I had to queue to know that, too. Nowhere to go and no flight to take, we decided to rent a car and head to Calais to get the ferry. We got a big car, put all the bags in and headed south.

Holland, Belgium and finally France, we got first to Dunkirk to check for ferries. There was only a very stupid lady that refused to answer me if there was a passenger ticket, just because I was not in the queue. Of course I would get the queue if there was, but what would be the point to get the whole queue if there wasn’t? Well, there wasn’t. Another fine lady answered me in 3 seconds, and it didn’t even hurt.

We headed to Calais, broken, hungry and without a single Euro, we could only find MacDonalds in Belgium and Holland. But we made it. After several U-turns along the way (the car got no satnav nor AVIS had any), we eventually got there. Now, it was just a matter of dropping the key back and get the 4 hours queue to buy our tickets to cross the channel…

As the sunset began, the cold that was already there began to show its face. The temperature dropped quickly, my hands froze and Renata and the kids went inside. No panic, as it was just a matter of waiting… how naive.

Reaching the tickets office, I checked passports, bags and wallet… wait, where’s my wallet?! Desperately looking in all bags, I called Renata, who kindly reminded me it was in the car’s arm pocket… But I had already gave the car keys already back, in the night safe.

When I got inside the building (3.5hours later), I ask them to stay in the queue and we decided to look for a way to get the wallet back. A French staff wanted to break in with a rock, but I reminded him there were hundreds of policemen around, British and French. It’d be harder to explain than to come home without the documents.

After almost an hour looking for a way to retrieve the key, with no avail, they decided to call the AVIS guy, who arrived after half hour, and he also wasn’t in his best mood… Well, who would? Finally, we made it to the boat, with all bags and children and documents… But our car was still parked on the Long Stay at Heathrow airport, 3 days due.

Thursday, 21st April 2010

When we finally reached Dover, we found that there were no cars to rent, buses were full and the next train would be only hours from then. To make matters worse, the only taxi big enough to carry us didn’t know where Cambridge was…

The Travel Centre in Dover was the key. Not only I could withdraw money (using my credit card, as my account was blocked), but someone gave me a list of telephones to call for a taxi. After a few attempts, one of them got the call and had a car big enough. We agreed on the price and he came to pick us up.

It took around 2 hours to get home, all of us (including the driver) trying to stay awake, but we finally got home around 3am, and after firing the usual emails (family, work, friends) to signal our arrival, we went to sleep.

After 2 days without sleeping (yes, I didn’t sleep a second), it was a miracle I could wake up at noon. We had a minimalistic breakfast (there was no food at all at home) and, without so much of a shower, I headed south, to Heathrow airport, via train.

The train was fine, but the underground train decided it didn’t want to go all the way to terminal 4, so it stopped at Northfields and spat everyone out. I got the next train, which also didn’t go to terminal 4, and politely asked us to leave to get the next train yet, which finally took us to terminal 4.

3 hours after I left home, I was inside our car. All working, I just forgot the ticket, which was easy to work around since I had the booking confirmation and they opened the gates for me. I was expecting some traffic on the M25, which came as sure as rain in England, but my tiredness was so great that I couldn’t stay awake for too long.

I stopped at the A1 services, and had a light meal and a quadruple-espresso that I mixed with a heavy-chocolate milkshake and swallowed in a few gulps. It was yet another hour or so, almost falling asleep, that I finally got home… after 7pm.

All of us, safe and sound, and I had all the time in the world to sleep comfortably in a nice bed. There was only one detail.. During all that adventure, both kids slept like angels through the long boring parts, so they weren’t tired at all!

It is only now, three days after we arrived, that I feel I’m getting back my energy.

Minix seems to be inspiring more operating systems nowadays. Microsoft Research is investing on a micro-kernel (they call it multi-kernel, as there are slight differences) called Barrelfish.

Despite being Microsoft, it’s BSD licensed. The mailing list looks pretty empty, the last snapshot is half a year ago and I couldn’t find an svn repository, but still more than I would expect from Microsoft anyway.

Multi-kernel

The basic concept is actually very interesting. The idea is to be able to have multi-core hybrid machines to the extreme, and still be able to run a single OS on it. Pretty much the same way some cluster solutions do (OpenMPI, for instance), but on a single machine. The idea is far from revolutionary. It’s a natural evolution of the multi-core trend with the current cluster solutions (available for years) and a fancy OS design (micro-kernel) that everyone learns in CS degrees.

What’s the difference, then? For one thing, the idea is to abstract everything away. CPUs will be just another piece of hardware, like the network or graphic cards. The OS will have the freedom to ask the GPU to do MP floating-point calculations, for instance, if it feels it’s going to benefit the total execution time. It’ll also be able to accept different CPUs in the same machine, Intel and ARM for instance (like the Dell Latitude z600), or have different GPUs, nVidia and ATI, and still use all the hardware.

With Windows, Linux and Mac today, you either use the nVidia driver or the ATI one. You also normally don’t have hybrid-core machines and absolutely can’t recover if one of the cores fail. This is not the same with cluster solutions, and Barrelfish’s idea is to simulate precisely that. In theory, you could do energy control (enabling and disabling cores), crash-recovery when one of the cores fail but not the other, or plug and play graphic or network cards and even different CPUs.

Imagine you have an ARM netbook that is great for browsing, but you want to play a game on it. You get your nVidia and a coreOcta 10Ghz USB4 and plug in. The OS recognizes the new hardware, loads the drivers and let you play your game. Battery life goes down, so once you’re back from the game, you just unplug the cards and continue browsing.

Scalability

So, how is it possible that Barrelfish can be that malleable? The key is communication. Shared memory is great for single-processed threaded code and acceptable for multi-processed OSs with little number of concurrent process accessing the same region in memory. Most modern OSs can handle many concurrent processes, but they rarely access the same data at the same time.

Normally, processes are single threaded or have a very small number of threads (dozens) running. More than that is so difficult to control that people usually fall back to other means, such as client/server or they just go out and buy more hardware. In clusters, there is no way to use shared memory. For one, accessing memory in another computer via network is just plain stupid, but even if you use shared memory in each node and client/server among different nodes, you’re bound to have trouble. This is why MPI solutions are so popular.

In Barrelfish there’s no shared memory at all. Every process communicate with each other via messages and duplicate content (rather than share). There is an obvious associated cost (memory and bus), but the lock-free semantics is worth it. It also gives Barrelfish another freedom: to choose the communication protocol generic enough so that each piece of hardware is completely independent of all others, and plug’n'play become seamless.

Challenges

It all seem fantastic, but there’s a long road ahead. First, message passing scales much better than shared memory, but nowadays there isn’t enough cores in most machines to make it worth it. Message passing also introduces some other problems that are not easily solvable: bus traffic and storage requirements increase considerably, and messages are not that generic in nature.

Some companies are famous for not adhering to standards (Apple comes to mind), and a standard hardware IPC framework would be quite hard to achieve. Also, even if using pure software IPC APIs, different companies will still use slightly modified APIs to suit their specific needs and complexity will rise, exponentially.

Another problem is where the hypervisor will live. Having a distributed control centre is cool and scales amazingly well, but its complexity also scales. In a hybrid-core machine, you have to run different instructions, in different orders, with different optimizations and communication. Choosing one core to deal with the scheduling and administration of the system is much easier, but leaves the single-point-of-failure.

Finally, going the multi-hybrid-independent style is way too complex. Even for a several-year’s project with lots of fund (M$) and clever people working on it. After all, if micro-kernel was really that useful, Tanembaum would have won the discussion with Linus. But, the future holds what the future holds, and reality (as well as hardware and some selfish vendors) can change. Multi-kernel might be possible and even easier to implement in the future.

This seems to be what the Barrelfish’s team is betting on, and I’m with them on that bet. Even if it fails miserably (as did Minix), some concepts could still be used in real-world operating systems (like Minix), whatever that’ll mean in 10 years. Being serious about parallelism is the only way forward, sticking with 40 years old concepts is definitely not.

I’m still aspiring for non-deterministic computing, though, but that’s an even longer shot…

A while ago I wrote an article about Agile and Scrum and wanted to write another one following my recent experience with Agile. However, somehow I couldn’t add anything of that great value to my original post that would be worth a new one.

And now I know I don’t have to. In this fantastic post, Gwaredd takes a deep look into all failures and successes of Agile, with the common misconceptions of believers and decision-makers. In the end, the so called “Post Agile”, is just plain common sense.

A long time ago I read an article about some dangerous psychological studies in the 70′s. It’s funny to think that, at that time, things that we don’t even consider doing, were acceptable.

Can you imagine yourself with a periscope counting the seconds some truck drivers take to piss in a public toilet? Or pretending to rape a girl and risk getting shot (especially in the US)? It’s not just ethically incorrect, it’s dangerous!

Recently, I read an article about some students monitoring 350 million mobile calls just to figure out if the callee’d call you back. Not only in the 70′s that would be nonsense, but people would explode in rage, as it’d be just enough to prove all conspiracy theories at that time (not to mention the cold war).

This is not the first research using “unnamed” data from carriers or websites, nor will be the last. I myself proposed something similar to Yahoo! when I worked there to get the trends and act on the average (rather than tag individuals), and I see now that it’s becoming acceptable to allow research groups to openly read entire databases that before was considered private.

I don’t particularly dislike such type of research, especially when they’re done by universities, but the slight paranoia feeling creep up my spine sometimes. I guess that’s one of the issues that is dividing people into two very distinctive groups: those that ignore completely the privacy for the sake of comfort, and those that ignore comfort for the sake of privacy.

I am in between the two groups, but I can’t say I’m exactly average. I think I’m an extremist on both sides. I don’t mind storing my private emails on Google but I disable all Facebook add-ons and restrict access to all my personal data. I pay everything on the internet with my credit-card but I’ll refuse to the end of my days to use the biometric passport or iris recognition at airports.

There is no logic, really, it’s just the kind of thing you stick with. It is true that governments have more power to dig your data when they want, while Amazon will probably only have my credit-card number. But it’s also true that no government in the world can dig everyone’s data all the time, so it’s pretty improbable that someone is monitoring how many times I cross the Heathrow border.

In the end, only one thing makes out as logic in the whole scene: during the recent years, it was far more likely the government loosing all banking details of everyone in the country than some hacker invading Amazon to get my credit-card. Maybe that’s what’s keeping me from accepting IDs and biometric passports… or maybe I never will…

« Previous entries