systemcall dot org

Minix seems to be inspiring more operating systems nowadays. Microsoft Research is investing on a micro-kernel (they call it multi-kernel, as there are slight differences) called Barrelfish.

Despite being Microsoft, it’s BSD licensed. The mailing list looks pretty empty, the last snapshot is half a year ago and I couldn’t find an svn repository, but still more than I would expect from Microsoft anyway.

Multi-kernel

The basic concept is actually very interesting. The idea is to be able to have multi-core hybrid machines to the extreme, and still be able to run a single OS on it. Pretty much the same way some cluster solutions do (OpenMPI, for instance), but on a single machine. The idea is far from revolutionary. It’s a natural evolution of the multi-core trend with the current cluster solutions (available for years) and a fancy OS design (micro-kernel) that everyone learns in CS degrees.

What’s the difference, then? For one thing, the idea is to abstract everything away. CPUs will be just another piece of hardware, like the network or graphic cards. The OS will have the freedom to ask the GPU to do MP floating-point calculations, for instance, if it feels it’s going to benefit the total execution time. It’ll also be able to accept different CPUs in the same machine, Intel and ARM for instance (like the Dell Latitude z600), or have different GPUs, nVidia and ATI, and still use all the hardware.

With Windows, Linux and Mac today, you either use the nVidia driver or the ATI one. You also normally don’t have hybrid-core machines and absolutely can’t recover if one of the cores fail. This is not the same with cluster solutions, and Barrelfish’s idea is to simulate precisely that. In theory, you could do energy control (enabling and disabling cores), crash-recovery when one of the cores fail but not the other, or plug and play graphic or network cards and even different CPUs.

Imagine you have an ARM netbook that is great for browsing, but you want to play a game on it. You get your nVidia and a coreOcta 10Ghz USB4 and plug in. The OS recognizes the new hardware, loads the drivers and let you play your game. Battery life goes down, so once you’re back from the game, you just unplug the cards and continue browsing.

Scalability

So, how is it possible that Barrelfish can be that malleable? The key is communication. Shared memory is great for single-processed threaded code and acceptable for multi-processed OSs with little number of concurrent process accessing the same region in memory. Most modern OSs can handle many concurrent processes, but they rarely access the same data at the same time.

Normally, processes are single threaded or have a very small number of threads (dozens) running. More than that is so difficult to control that people usually fall back to other means, such as client/server or they just go out and buy more hardware. In clusters, there is no way to use shared memory. For one, accessing memory in another computer via network is just plain stupid, but even if you use shared memory in each node and client/server among different nodes, you’re bound to have trouble. This is why MPI solutions are so popular.

In Barrelfish there’s no shared memory at all. Every process communicate with each other via messages and duplicate content (rather than share). There is an obvious associated cost (memory and bus), but the lock-free semantics is worth it. It also gives Barrelfish another freedom: to choose the communication protocol generic enough so that each piece of hardware is completely independent of all others, and plug’n'play become seamless.

Challenges

It all seem fantastic, but there’s a long road ahead. First, message passing scales much better than shared memory, but nowadays there isn’t enough cores in most machines to make it worth it. Message passing also introduces some other problems that are not easily solvable: bus traffic and storage requirements increase considerably, and messages are not that generic in nature.

Some companies are famous for not adhering to standards (Apple comes to mind), and a standard hardware IPC framework would be quite hard to achieve. Also, even if using pure software IPC APIs, different companies will still use slightly modified APIs to suit their specific needs and complexity will rise, exponentially.

Another problem is where the hypervisor will live. Having a distributed control centre is cool and scales amazingly well, but its complexity also scales. In a hybrid-core machine, you have to run different instructions, in different orders, with different optimizations and communication. Choosing one core to deal with the scheduling and administration of the system is much easier, but leaves the single-point-of-failure.

Finally, going the multi-hybrid-independent style is way too complex. Even for a several-year’s project with lots of fund (M$) and clever people working on it. After all, if micro-kernel was really that useful, Tanembaum would have won the discussion with Linus. But, the future holds what the future holds, and reality (as well as hardware and some selfish vendors) can change. Multi-kernel might be possible and even easier to implement in the future.

This seems to be what the Barrelfish’s team is betting on, and I’m with them on that bet. Even if it fails miserably (as did Minix), some concepts could still be used in real-world operating systems (like Minix), whatever that’ll mean in 10 years. Being serious about parallelism is the only way forward, sticking with 40 years old concepts is definitely not.

I’m still aspiring for non-deterministic computing, though, but that’s an even longer shot…

A while ago I wrote an article about Agile and Scrum and wanted to write another one following my recent experience with Agile. However, somehow I couldn’t add anything of that great value to my original post that would be worth a new one.

And now I know I don’t have to. In this fantastic post, Gwaredd takes a deep look into all failures and successes of Agile, with the common misconceptions of believers and decision-makers. In the end, the so called “Post Agile”, is just plain common sense.

A long time ago I read an article about some dangerous psychological studies in the 70’s. It’s funny to think that, at that time, things that we don’t even consider doing, were acceptable.

Can you imagine yourself with a periscope counting the seconds some truck drivers take to piss in a public toilet? Or pretending to rape a girl and risk getting shot (especially in the US)? It’s not just ethically incorrect, it’s dangerous!

Recently, I read an article about some students monitoring 350 million mobile calls just to figure out if the callee’d call you back. Not only in the 70’s that would be nonsense, but people would explode in rage, as it’d be just enough to prove all conspiracy theories at that time (not to mention the cold war).

This is not the first research using “unnamed” data from carriers or websites, nor will be the last. I myself proposed something similar to Yahoo! when I worked there to get the trends and act on the average (rather than tag individuals), and I see now that it’s becoming acceptable to allow research groups to openly read entire databases that before was considered private.

I don’t particularly dislike such type of research, especially when they’re done by universities, but the slight paranoia feeling creep up my spine sometimes. I guess that’s one of the issues that is dividing people into two very distinctive groups: those that ignore completely the privacy for the sake of comfort, and those that ignore comfort for the sake of privacy.

I am in between the two groups, but I can’t say I’m exactly average. I think I’m an extremist on both sides. I don’t mind storing my private emails on Google but I disable all Facebook add-ons and restrict access to all my personal data. I pay everything on the internet with my credit-card but I’ll refuse to the end of my days to use the biometric passport or iris recognition at airports.

There is no logic, really, it’s just the kind of thing you stick with. It is true that governments have more power to dig your data when they want, while Amazon will probably only have my credit-card number. But it’s also true that no government in the world can dig everyone’s data all the time, so it’s pretty improbable that someone is monitoring how many times I cross the Heathrow border.

In the end, only one thing makes out as logic in the whole scene: during the recent years, it was far more likely the government loosing all banking details of everyone in the country than some hacker invading Amazon to get my credit-card. Maybe that’s what’s keeping me from accepting IDs and biometric passports… or maybe I never will…

Ever since 1995 I hear the same phrase, and ever since 2000 I stopped listening. It was already the year of Linux in 95 for me, so why bother?

But this year is different, and Linux is not the only revolution in town… By the end of last year, the first tera-electronvolt collisions were recorded in the LHC, getting closer to see (or not) the infamous Higgs boson. Now, the NIF reports a massive 700 kilojoules in a 10 billionth of a second laser, that, if it continues on schedule, could lead us to cold fusion!!

The human race is about to finally put the full stop on the standard model and achieve cold fusion by the end of this year, who cares about Linux?!

Well, for one thing, Linux is running all the clusters being used to compute and maintain all those facilities. So, if it were for Microsoft, we’d still be in the stone age…

UPDATE: More news on cold fusion…

There was a time when the gods were bored to death, but because they couldn’t actually die, they started writing down all their knowledge to pass the time. Virtually everything known to them was written on the ancient manuscripts and organized by topic, cross-linked with other topics, in a very simple yet complete language that described everything to the last detail.

Time passed, universes were created and in some of them, creatures developed critical thinking. With that, came science and with science, the logical conclusion that gods didn’t actually have to exist was inevitable. So inevitable that finally, without delay, the gods actually died. For the curious minds, that fact is based on the quantum principle that, if no one sees it, it doesn’t actually exist.

Yet, for the great benevolence of the Universe, the manuscripts were kept and for billions of (Earth) years (relativistically speaking, of course), they were forgotten. But everything that is lost is waiting to be found, and in a very small speck of dust, in a completely irrelevant galaxy on the (multi-dimensional) margins of one of the universes, a yet-to-be intelligent race found a way to the manuscripts. However, their intelligence was not enough to uncover the whole truth. They could only gather hints and pieces of what once was the complete knowledge of everything.

It was much more of a coincidence, really, that so many of those beings would channel the truth through their fingers and type them, guided by the manuscripts themselves, on a remote system that all the other beings would go and search for knowledge. Some would misguide them, of course, and others would fight over the truth, for no one really know how to interpret such manuscripts. Seeing such confusion and regress, the Universe decided to create a game, on which such frivolous beings could channel their good side instead, even if not consciously knowing so.

The Game

The game is very simple and is meant to beings with very limited mental and social capacity.

The younger member starts by clicking on the “Random Article” link on any Wikipedia page, or by choosing a subject from the main page. After that, the following rules must be repeated until the players are tired or bored to death:

1. The current player must explain (out loud) what the article is about and think of a related subject. The relation can be of any kind.
2. The other players would then decide if the relation is valid and the player should then go to the related page.
3. If the relationship is valid and approved, the points are counted on the following manner:
   • 1 point if the article exists, +3 points if the player enhances it.
   • 3 points if the article doesn’t exist, +9 points if the player creates it with a stub.
4. The player on the right goes next.

Of course, at least one access to Wikipedia is necessary, but many can be used simultaneously. It is considered foul play to tamper with the contents of the pages just to get extra points (remember, the gods won’t be pleased at all!).

In between games, there is a way to get extra points for the next round. If the player proves that he/she enhanced Wikipedia pages quoted from a previous game (change logs suffice), he/she gets +3 points at the start of the next game for every considerable change (10 or more words) in a single page. Multiple changes in the same page counts as one change and the points can only be counted if the change happened between the last game and current, so the same change cannot be considered twice. Creation of new pages related to the subjects mentioned also count as change.

The Winner

The winner of the game is obviously the one that gets most points, but the real winner is the society. Knowledge has no owner, no boundaries, no limits. The more you share, the more society benefits. Knowledge is power, and you can give it for free, as easy as writing an email… to the world.

Most game-changing scientific discoveries had a lot of logic and critical thinking, but also a bit of luck involved. As most scientists, I don’t believe in luck, so the definition of luck here is being the right person in the right place at the right time. As most (good) scientists, I don’t believe, I state, hypothesise, prove, refute, so the definition of belief here is also obvious.

My point is that evolution wouldn’t have been formulated if Darwin hadn’t gone with the Beagle, genetics wouldn’t be so solid if Mendel hadn’t believed the contrary so fiercely, Plank wouldn’t have found the quantum if there wasn’t a major argument about the black-body spectrum and Einstein would have won the Nobel prize for any other thing if he hadn’t been so drawn by God playing dice.

My story today starts in a similar way, but in a much more mundane problem… I lost my keys.

There is nothing I hate more than loosing my keys, especially in the 25th of December when we’re going to hit the road in the 27th. I lost all my keys, car, house, even my USB key. These modern car keys are not easy to replicate, I’d have to buy the whole thing again and loosing your front door key is not the kind of thing you let pass with a simple copy, you have to change the whole set, especially when you’re going away for a week.

Well, after despair came fear. After fear, despair again. We searched the whole house, inside, outside and in between. Nothing. Brute force wasn’t helping, but that hadn’t stopped me to do it once in a while again, just in case. In between the despair brute-force moments, we decided to be logical about the situation and think, rather than search for the answer.

First point, we had a spare of either car and house, so at least we could still travel and come back home. My worries were, in fact, what would we find when we came back home… If I had lot my keys outside or had left them hanging off the front door’s key hole (happened more than once), it’d be just too easy for someone to clean the house while we were away.

So we tracked down every place we went, every thing we did. By logic, I couldn’t have lost them in the city or anywhere I would have gone by car. Nor I could have lost it inside the car, so at least we knew that it’d be either inside the house or around it (including the key hole, unfortunately). I almost cancelled our trip because of the key hole probability, but Renata, very logically, convinced me that everything we did could not have caused me to leave it there. It was very, very unlikely. So we went…

However very unlikely, that still bugged me the whole week and I felt a bit of panic when we got home. But to my comfort, the house was exactly the way we left. That was, in a twisted way, another indication that the key was not left in the key hole. It had to be inside the house. I went back to work, still using the spare keys, but always thinking about it, wondering wherever it was. Sometimes, just in case, I’d imagine that I would look somewhere and see the key there, and be very surprised I haven’t seen it there before. That feeling never came.

This week I thought enough was enough. I had to continue with my life, change the front door keys and buy the very expensive key set from the car’s manufacturers. I put a to-do in my mobile: “call toyota, landlord wrt keys”. It was then that luck stroke with an impeccable logic. I felt like Darwin finding the platypus or Mendel smashing peas.

I looked at our bag of snow jackets, hermetically sealed for the next winter (Cambridge has only one chance of snowing each year, and that was before Christmas), and thought: “If the keys are in there, we’ll only find out next winter.” The simple logic led me to think it’d be much cheaper for me to re-open the impossible-to-close-hermetically-sealed bag now and not find the key than to wait until next winter and have spent thousands of pounds for nothing. The risk assessment was positive, and that led me to the next piece of information that closed the gap: it was snowing before Christmas! It had to be there!

I opened the bag and tapped my jacket, nothing. But the logic was impeccable, I couldn’t be wrong. I wore the jacket and trusted logic above my own despair. Gently sliding my hands inside the pockets, as I always do. The pockets are deep, and I felt nothing at start, but that didn’t stop my trust in logic. Spock would have laughed at me if I did, it’s that serious, a vulcan could actually laugh. It was not out of faith or belief, it was the ultimately trust that scientists lay on logic above all feelings, common sense and general knowledge, that kept me going until I finally felt something…

I have recently joined the IETF Smart Grid group to see what people were talking about it and to put away my fears on security and privacy. What I saw was a bunch of experts discussing the plethora of standards that could be applied (very important) but few people seemed too interested in the privacy issue.

If you see the IEEE page on Smart Grids, besides the smart generation / distribution / reception (very important) there is a paragraph on the interaction between the grid and the customers, being very careful not to mention invasive techniques to allow the grid to control customer’s appliances:

“Intelligent appliances capable of deciding when to consume power based on pre-set customer preferences.”

Here, they focus on letting the appliances decide what will be done to save power, not the grid or the provider. Later on, on the same paragraph:

“Early tests with smart grids have shown that consumers can save up to 25% on their energy usage by simply providing them with information on that usage and the tools to manage it.”

Again, enforcing that the providers will only “provide [the customer] with information”. In other words, the grid is smart up to the smart meter (that is controlled by the provider), where inside people’s houses, it’s the appliances that have to be smart. One pertinent comment from Hector Santos in the IETF group:

“Security (most privacy) issues, I believe, has been sedated over the years with the change in consumer mindset. Tomorrow (and to a large extent today) generation of consumers will not even give it a second thought. They will not even realize that it was once considered a social engineering taboo to conflict with user privacy issues.”

I hate to be pessimist, but there is a very important truth in this. Not only people are allowing systems to store their data for completely different reasons, but they don’t care if the owner of the system will distribute their information or not. I, myself, always paranoid, have signed contracts with providers knowing that they would use and sell my data to third parties. The British Telecom is one good example. He continues:

“Just look how social networking and the drive to share more, not less has changed the consumer mindset. Tomorrow engineers will be part of all this new mindset.”

There is no social engineering any more like it used to be. Who needs to steal your information when it’s already there, on your Facebook? People are sharing willingly, and a lot of them know what problems it may cause, but the benefit, for them, is greater. Moreover, millions bought music, games and films with DRM, allowing a company control what you do, see or listen. How many Kindles were bought? How many iPhones? People don’t care what’s going on if they have what they want.

That is the true meaning of sedated privacy concerns. It’s a very distorted way of selfishness, where you don’t care about yourself, as long as you are happy. If it makes no sense to you, don’t worry, it makes no sense to me too.

Recently, the Future of Privacy Forum published an excellent analysis (via Ars) on the smart grid privacy. Several concepts that are easy to understand how dangerous they can be, became commonplace to not think about it or even consider it a silly worry, given that no one cares anyway.

An evil use of a similar technology is the “Selectable Output Control“. Just like a Kindle, the media companies want to make sure you only watch what you pay for. It may seem fair, and even cheaper, as they allow “smart pricing”, like some smart-grid technologies.

But we all have seen what Amazon did to kindle users, of Apple did to its AppStore, taking down contents without warn, removing things you paid for from your device, allowing or disallowing you to run applications or contents on your device as if you hadn’t pay enough money to own the device and its contents.

In the end, “smart pricing” is like tax cut, they reduce tax A, but introduce taxes B, C and D, which double the amount of taxes you pay. Of course, you only knew about tax A and went happy about your life. All in all, nobody cares who or how much they pay, as long as they can get the newest fart app…

Star trek seems a long way and yet, a few news had made into the headlines exposing some achievements that might lead us closer to Roddenberry’s universe.

Some research just found anti-matter in an unusual place: lightning! It might be easier to produce a warp core that we originally thought. Given, of course, that sub-space exists and can be reached by an matter/anti-matter reaction.

Another research, from the University of California, has just found a way to create a medical tricorder. That, for me, is the best achievement so far. Not to mention time travels, teleportation, quantum computers and faster-than-light communication already achieved since the series was created.

Finally, the University of Canada just made the first phaser. Though, it’s still only set to stun…

But I have to say that I’m a bit worried. The Temporal Prime Directive might be needed a bit sooner than the 29th century…

Normally the Linux Magazine has great articles. Impartial, informative and highly technical. Unfortunately, not always. In a recent article, some perfectionist zealot stated that Ubuntu makes Linux looks bad. I couldn’t disagree more.

Ubuntu is a fast-paced, fast-adapted Linux. I was one of the early adopters and I have to say that most of the problems I had with the previous release were fixed. Some bugs went through, of course, but they were reported and quickly fixed. Moreover, Ubuntu has the support from hardware manufacturers, such as Dell, and that makes a big difference.

Linux is everything

Linux is excellent for embedded systems, great for network appliances, wonderful for desktops, irreplaceable as a development platform, marvellous on servers and the only choice for real clusters. It also sucks when you have to find the configuration manually, it’s horrible to newbies, it breaks whenever a new release is out, it takes longer to get new software (such as Firefox) but also helps a lot with package dependencies. Something that neiter Mac nor Windows managed to do properly over the past decades.

Linux is great as any piece of software could be but horrible as every operating system that was release since the beginning of times. Some Linux distributions are stable, others not so. Debian takes 10 years to release and when it does, the software it contains is already 10 years old. Ubuntu tries to be a bit faster but that obviously breaks a few things. If you’re fast enough fixing, the early adopters will be pleased that they helped the community.

“Unfortunately what most often comes is a system full of bugs, pain, anguish, wailing and gnashing of teeth – as many “early” adopters of Karmic Koala have discovered.”

As any piece of software, open or closed, free or paid, free or non-free. It takes time to mature. A real software engineer should know better, that a system is only fully tested when it reaches the community, the user base. Google uses their own users (your granny too!) as beta testers for years and everyone seem to understand it.

Debian zealots hate Red Hat zealots and both hate Ubuntu zealots that probably hate other zealots anywhere else. It’s funny to see how opinions vary greatly from a zealot clan to the other about what Linux really is. All of them have a great knowledge on what Linux is comprised of, but few seems to understand what Linux really is. Linux, or better, GNU/Linux is a big bunch of software tied together with so many different points of view that it’s impossible to state in less than a thousand words what it really is.

“Linux is meant to be stable, secure, reliable.”

NO, IT’S NOT! Linux is meant to be whatever you make of it, that’s the real beauty. If Canonical thought it was ready to launch is because they thought that, whatever bug pased the safety net was safe enough for the users to grab and report, which we did! If you’re not an expert, wait for the system to cool down. A non-expert will not be an “early adopter” anyway, that’s for sure.

Idiosyncrasies

Each Linux has its own idiosyncrasies, that’s what makes it powerful, and painful. The way Ubuntu updates/upgrades itself is particular to Ubuntu. Debian, Red Hat, Suse, all of them do it differently, and that’s life. Get over it.

“As usual, some things which were broken in the previous release are now fixed, but things which were working are now broken.”

One pleonasm after another. There is no new software without new bugs. There is no software without bugs. What was broken was known, what is new is unknown. How can someone fix something they don’t know? When eventually the user tested it, found it broken, reported, they fixed! Isn’t it simple?

“There’s gotta be a better way to do this.”

No, there isn’t. Ubuntu is like any other Linux: Like it? Use it. Don’t like it? Get another one. If you don’t like the way Ubuntu works, get over it, use another Linux and stop ranting.

Red Hat charges money, Debian has ubber-stable-decade-old releases, Gentoo is for those that have a lot of time in their hands, etc. Each has its own particularities, each is good for a different set of people.

Why Ubuntu?

I use Ubuntu because it’s easy to install, use and update. The rate of bugs is lower than on most other distros I’ve used and the rate of updates is much faster and stable than some other distros. It’s a good balance for me. Is it perfect? Of course not! There are lots of things I don’t like about Ubuntu, but that won’t make me use Windows 7, that’s for sure!

I have friends that use Suse, Debian, Fedora, Gentoo and they’re all as happy as I am, not too much, but not too few. Each has problems and solutions, you just have to choose the ones that are best for you.

Gtk, the graphical interface behind Gnome, is very simple to use. It doesn’t have an all-in-one IDE such as KDevelop, which is very powerful and complete, but it features a simple and functional interface designer called Glade. Once you have the widgets and signals done, filling the blanks is easy.

As an example, I wrote a simple dice throwing application, which took me about an hour from install Glade to publish it on the website. Basically, my route was to apt-get install glade, open it and create a few widgets, assign some callbacks (signals) and generate the C source code.

After that, the file src/callbacks.c contain all the signal handlers to which you have to implement. Adding just a bit of code and browsing this tutorial to get the function names was enough to get it running.

Glade generates all autoconf/automake files, so it was extremely easy to compile and run the mock window right at the beginning. The rest of the code I’ve added was even less code than I would add if doing a console based application to do just the same. Also, because of the code generation, I was afraid it’d replace my already changed callbacks.c when I changed the layout. Luckily, I was really pleased to see that Glade was smart enough not to mess up with my changes.

My example is not particularly good looking (I’m terrible with design), but that wasn’t the intention anyway. It’s been 7 years since the last time I’ve built graphical interfaces myself and I’ve never did anything with Gtk before, so it shows how easy it is to use the library.

Just bear in mind a few concepts of GUI design and you’ll have very little problems:

1. Widget arrangement is not normally fixed by default (to allow window resize). So workout how tables, frames, boxes and panes work (which is a pain) or use fixed position and disallow window resize (as I did),
2. Widgets don’t do anything by themselves, you need to assign them callbacks. Most signals have meaningful names (resize, toggle, set focus, etc), so it’s not difficult to find them and create callbacks for them,
3. Side effects (numbers appearing at the press of a button, for instance) are not easily done without global variables, so don’t be picky on that from start. Work your way towards a global context later on when the interface is stable and working (I didn’t even bother)

If you’re looking for a much better dice rolling program for Linux, consider using rolldice, probably available via your package manager.

« Previous entries