UFS & ZFS in Virtual Machine vs Ext3 Physical File Benchmarks


Some really bizarre results here that just blew my mind. Check out the “rate” column for both random writes and random reads and compare the Ext3 Ubuntu with Nexenta VM for both UFS and ZFS – UFS is a bit quicker the ZFS. I am not sure whether the virtualisation actually improves the random reads and writes but it’s a pretty massive difference. After my small play with Nexenta, it’s looking quite promising to become my primary desktop OS.

Here’s a tiobench of Ubuntu 7.04 running on a Seagate ST3320620AS 320GB SATA drive.

Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
2.6.20-16-generic 1792 4096 1 54.81 9.103% 0.070 370.19 0.00000 0.00000 602
2.6.20-16-generic 1792 4096 2 44.26 13.23% 0.175 791.39 0.00000 0.00000 334
2.6.20-16-generic 1792 4096 4 49.33 31.25% 0.306 1187.43 0.00000 0.00000 158
2.6.20-16-generic 1792 4096 8 47.55 59.80% 0.613 1626.75 0.00000 0.00000 79

Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
2.6.20-16-generic 1792 4096 1 0.51 0.771% 7.648 136.11 0.00000 0.00000 66
2.6.20-16-generic 1792 4096 2 0.52 1.272% 15.015 155.48 0.00000 0.00000 41
2.6.20-16-generic 1792 4096 4 0.55 2.649% 27.618 572.07 0.00000 0.00000 21
2.6.20-16-generic 1792 4096 8 0.55 5.843% 53.622 1032.24 0.00000 0.00000 9

Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
2.6.20-16-generic 1792 4096 1 32.42 17.40% 0.116 5364.25 0.00109 0.00000 186
2.6.20-16-generic 1792 4096 2 45.88 58.74% 0.158 4584.56 0.00044 0.00000 78
2.6.20-16-generic 1792 4096 4 43.31 105.3% 0.325 4418.34 0.00196 0.00000 41
2.6.20-16-generic 1792 4096 8 41.73 198.6% 0.654 6986.02 0.00763 0.00000 21

Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
2.6.20-16-generic 1792 4096 1 2.00 0.972% 0.257 15.78 0.00000 0.00000 206
2.6.20-16-generic 1792 4096 2 1.92 2.064% 0.503 150.80 0.00000 0.00000 93
2.6.20-16-generic 1792 4096 4 1.59 3.425% 1.691 332.46 0.00000 0.00000 47
2.6.20-16-generic 1792 4096 8 1.78 6.911% 1.178 546.28 0.00000 0.00000 26

Now – Running inside the same physical machine in a Virtual Machine using Vmware Server Version 1.0.3-1 I am running SunOS sun1 5.10 NexentaOS_20070402 i86pc i386 i86pc Solaris with all ‘apt-get upgrade’ applied as of today.

The vmdk files live on the same physical hard drive of the above benchmark tests and obviously the benchmarks were not being performed at the same time.

First, the UFS volume, living at /

Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 69.05 51.46% 0.055 280.48 0.00000 0.00000 134
5.10 512 4096 2 110.90 157.3% 0.065 280.11 0.00000 0.00000 70
5.10 512 4096 4 113.98 318.0% 0.120 390.12 0.00000 0.00000 36
5.10 512 4096 8 127.42 715.1% 0.214 725.61 0.00000 0.00000 18

Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 51.53 45.93% 0.073 154.57 0.00000 0.00000 112
5.10 512 4096 2 55.09 82.72% 0.089 97.44 0.00000 0.00000 67
5.10 512 4096 4 102.26 98.82% 0.036 0.90 0.00000 0.00000 103
5.10 512 4096 8 121.49 96.99% 0.029 0.24 0.00000 0.00000 125

Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 37.51 33.65% 0.097 890.04 0.00000 0.00000 111
5.10 512 4096 2 87.13 113.1% 0.077 510.31 0.00000 0.00000 77
5.10 512 4096 4 77.77 195.4% 0.154 604.28 0.00000 0.00000 40
5.10 512 4096 8 89.91 446.0% 0.283 924.70 0.00000 0.00000 20

Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 3.39 25.89% 1.123 180.35 0.00000 0.00000 13
5.10 512 4096 2 12.12 72.10% 0.366 509.51 0.00000 0.00000 17
5.10 512 4096 4 10.33 93.73% 0.786 340.37 0.00000 0.00000 11
5.10 512 4096 8 15.49 299.5% 1.254 428.83 0.00000 0.00000 5

and here’s the home directory, which is running ZFS

Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 128.28 96.71% 0.030 36.40 0.00000 0.00000 133
5.10 512 4096 2 127.90 188.5% 0.056 200.39 0.00000 0.00000 68
5.10 512 4096 4 130.20 360.9% 0.108 380.10 0.00000 0.00000 36
5.10 512 4096 8 130.67 689.6% 0.211 640.09 0.00000 0.00000 19

Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 74.75 71.35% 0.043 17.40 0.00000 0.00000 105
5.10 512 4096 2 112.29 99.28% 0.032 1.29 0.00000 0.00000 113
5.10 512 4096 4 105.33 204.9% 0.074 92.12 0.00000 0.00000 51
5.10 512 4096 8 125.68 98.95% 0.028 0.23 0.00000 0.00000 127

Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 99.39 67.10% 0.034 569.68 0.00000 0.00000 148
5.10 512 4096 2 86.28 118.8% 0.066 558.60 0.00000 0.00000 73
5.10 512 4096 4 76.45 201.4% 0.121 760.74 0.00000 0.00000 38
5.10 512 4096 8 108.75 558.3% 0.229 570.08 0.00000 0.00000 19

Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
—————————- —— —– — —— —— ——— ———– ——– ——– —–
5.10 512 4096 1 13.72 56.60% 0.263 49.72 0.00000 0.00000 24
5.10 512 4096 2 12.34 93.16% 0.558 282.44 0.00000 0.00000 13
5.10 512 4096 4 13.88 183.4% 0.920 350.58 0.00000 0.00000 8
5.10 512 4096 8 16.86 288.1% 1.027 510.47 0.00000 0.00000 6

Now, if I was doing a Masters or Phd, I would run these >5 times and take the average to make sure my results were accurate but I just ain’t got the time and no-ones publishing or referencing these little SATA disks, are they. If I can get the chance to install Nexenta on an ESX server that’s also running Suse 10.1, I’d like to see some vm->vm comparisons to see the performance difference between the two.

Anyway, Nexenta is looking ubercool ie: in it’s most basic form, think of being able to bolt on ZFS, Dtrace, Zones and Containers to Ubuntu and you have.. Nexenta – well, almost. I have been waiting for this ever since I read the announcement that Sun were open sourcing Solaris. A huge kudos to the guys over at Nexenta who give me Z-raided fantasies.

Powered by ScribeFire.

Technorati Tags: , , , ,

Links for the day – Javascript for Rich Internet Applications

Some Interesting Javascript – I am no JS guru, so comments are welcome

ExtJS – an interesting javascript library with some half-decent documentation and community, as well as paid support and alternative licensing

Shortcut.js – Keyboard enable your web application – this is something that’s seriously needed. I have seen bank tellers migrate from Terminal based systems to browser based systems where they were literally forced to work at a quarter of the speed of their terminal apps because the browser forces them into hunt and peck mode. Especially for old grannies who’s eyes were starting to fail them it can become painful to watch.

Oh – totally off topic but how to kill 5000 print jobs from your LPD queue in HP-UX?

cancel -e <printer_name>

I’ll have to look for the Linux alternative because the cancel command is deprecated in Linux for standard lp commands.

Technorati Tags: , ,

Powered by ScribeFire.

The Zen of Django


Not much time to write but all I can say about Django is:

I came for the models, I stayed for the templates

While it’s not all been smooth sailing for me mostly due to my lack of python experience, the guys on IRC especially “Magus” have been tremendous. Is he paid to help people??? Someone should give him a fulltime position on the IRC channel.

The other part that tripped me up was newforms library as it’s still under development and isn’t fully documented as yet. This article at Big Nerd Ranch was a big help as was Colin Grady, once more. I often found myself overcomplicating issues that were much simpler than I was imagining.

Anyway – the more I use Django, the more I learn the true depths of it’s greatness even though I still feel as though I am skimming the surface. I have met Malcolm Treddinick face to face, a fellow Australian and mad hacker. His commits on the Django project have been somewhat impressive. I can’t wait to be able to add some value back into the community once I get over a few more stumbling blocks.

Powered by ScribeFire.

Django screencasts on Showmedo

Have been busy working on a project in Django, so haven’t able to write as much as I like but:

Siddhi has some nice Django screencasts over at Showmedo.com

I was also reading the Trac / Babel website which ended up hooking me up to a great article from a guy who was thinking about succesful open source projects and came to the conclusion to how powerful api’s are in both open source and proprietary applications. The article pointed out how much having a good, documented api lowers the entry barrier for contributors, allowing a project to gain a community and momentum faster. I see this as part of the success of the Joomla project. People were able to go away and create modules, components and bots without having to submit or beg for features on the main project’s repo. I wish I could find the article now though!

Anyway, that led me to another worthy mention: Django Restful Interfaces project.

Beyond the django / python world, if you are into learning languages, I stumbled across Mango Languages today, which I am going to add to my use of Granule Flashcards for Spanish.

Technorati Tags: , , , ,

Powered by ScribeFire.

Open Source PDF Editor – Part 2


In my last post, I wrote about flpsed a PDF and Postscript editor. I was really surprised about how much traffic the post brought in from google, so I thought I would do a bit more digging around to see what I could find because flpsed doesn’t have the same amount of features as something like Foxit’s PDF Editor.

So I went on a bit of an install frenzy searching for Windows PDFEditor’s with free trials and installing them and running them with Wine. I didn’t have much success there and was about to try and install PDFEditor under wine because Frankscorner reports that PDFEditor run’s under wine.

But before I did, I managed to stumble across PDFEdit, an open source linux pdf editor with a very similar feature set to Foxit’s PDFEditor – god knows how come it took me so long to find it.

So, the next step was to get it installed right? It’s not in Ubuntu’s repo’s – it’s a QT app. So, I tried compiling from source but it complained that some of my libraries were too old and this is in Feisty Fawn?

So, I noticed an RPM version PDFedit 0.3.1 in sourceforge – ha ha.. let’s see if I can installed that after converting it to deb with alien.

Well, it worked and works brilliantly.

So, here’s the deal.

Download pdfedit-0.3.1-1.i386.rpm

sudo alien pdfedit-0.3.1-1.i386.rpm

Not sure why it creates it as pdfedit_0.3.1-2_i386.deb but anyway.

dpkg -i install pdfedit_0.3.1-2_i386.deb

and you’re ready to go. Of course, if you are on an RPM based distro, then I guess you could skip the alien step and try:

rpm -Uvh pdfedit-0.3.1-1.i386.rpm

or use smart.

Technorati Tags: , , ,

Powered by ScribeFire.

Returning to Amarok and Yakuake instead of Listen and Tilda??

Ok, I have used most of the Gnome music players and they are mostly very good and quite stable but here, I find myself back on Amarok, given up on it’s Gnome counterparts?

Well, mostly it’s the tagging capabilities that Amarok offers. Yes, Listen and a few other have musicbrainz support but most of my music is so left of field that it’s not on Musicbrainz and this is where Amarok let’s you use “guess tag from filename” and you can also do this quickly and easily for groups of files. Plus I discovered Amarok has “smart playlist” which works in a similar way that Listen does by selecting songs it thinks you are going to like from your music collection. I am not sure what action it uses to do this but it seems to work pretty good so far.

Now.. why Yakuake over Tilda? Well, there’s a few reasons for this.

  1. While running ‘top’ in a Yakuake window, if you drop it hide it and reshow, it causes top to core dump.
  2. Try using gnu ‘screen’ for split windows in Tilda and if you hide it and reshow, your (screen) split window is gone.. kaput.
  3. There’s no way to speed up the drop down beyond 150ms in Tilda, which makes it not so good when using NX for remote desktop. Yakuake can have a 0s delay, so it’s on or off.

and I still haven’t found a photo management application that I like. The closest is Digikam (another KDE app) because it gives me better access to the underlying file system than most other apps BUT I love the export options in F-spot Photo Manager – including flickr, picasaweb or smugmug. Digikam seems to be the most comprehensive and gives me a great deal more flexibility on what information I want to see and how I want to see it. F-spot is just sexier and sex appeal goes a long way for me. I am mostly using F-spot and a bit of Digikam but occasionally open Picasa (using wine) but I still haven’t found one that really grabs me.

So, I really am having a hot/cold relationship with Gnome apps at the moment. The only native gnome app I really use is Evolution and Nautilus but I still find myself missing the ability to use tabs. Yes, I have installed PCMan File Manager but it’s not so well integrated as Nautilus is with Gnome though I recently saw a post on Lifehacker that showed how to replace Nautilus with PCMan.. when I get time maybe.

Technorati Tags: , , , , , , , ,

Powered by ScribeFire.

Open Source Mind Mapping & Microfinance Software

I had seen, used and explored Freemind before but never REALLY put it to use. Recently, I began brainstorming for a project that I have had in my head for the last 12 months and I have found Freemind to be brilliant.

As is the case with most software I seek out, it is cross-platform and it’s written in Java. It has excellent shortcut capabilities that allow you to link, navigate, add, move, raise, lower, highlight etc all without touching the keyboard.. it really stands back, gets out of your way and lets the thoughts flow. It has the ability to export to html, sxw (old openoffice format), png or jpg format.

I was recently in Canberra attending a VMware course and I was able to connect to my home Linux server during my breaks and quickly add a few ideas to Freemind. I’ve even started using it to transfer my brainstorming into a database design so I can then start creating my django models.

So that’s freemind.

The next application I came across in my travels was Mifos ie MIcroFinance Open Source, a project of the Grameen foundation for managing loans and savings accounts for Microfinance institutions. It is built upon Java and Mysql although it aims to be database agnostic.

If you have never heard of the Grameen bank, it was founded by Muhammad Yunus, 2006 Nobel Peace prize winner who started providing Microloans to very poor (I dislike that term) people in Bangladesh in 1976. Since then, the Grameen Bank has flourished and has now issued more than US$ 5.1 billion to 5.3 million borrowers since it’s inception. Muhammad has also authored “Banker to the Poor: Micro-Lending and the Battle Against World Poverty” which is currently on my “to read” list.

So, there’s two great FREE applications. Free your mind or free people.. you choose.

Powered by ScribeFire.

Links of the day – for my own reference

I shall replicate these at the serious student when I revive it so as to keep this blog focused on open source. The serious student can deal with more life, living, art & philosophy as per the original intention a number of years ago.

Building Low Cost Disaster Resistant Housing Programs with Women – at changemakers.net and this link at worldchanging.com

Adobe buildings – Auroville Earth Building Insitute

Kiva – Direct p2p microfinancing

Ecolonomics – expressing the understanding that our ecology and economy are two sides of the same coin

For those that do arrive at this page, don’t let the “Maverick” title and the cheesy photo on the front cover of this book turn you off the content. I strongly believe this is the way that not only businesses should be run but also governments.

Powered by ScribeFire.

Open Source everywhere I turn


Is it just me or is this the year of open source? That ever elusive “year of Linux” may not arrive but certainly, it seems the virus of open source is taking hold. It certainly is happening in the kind of Web2.0 projects we are seeing coming up, and many businesses are choosing open business models of traditional models. Green is in, Green is good and open source is along for the ride. Or is it that suddenly because I am kind of focused on Open Source technologies, it starts sticking it’s head out at me from everywhere I look? I am truly being amazed at what’s happening in the world of open source, public domain, creative commons.. call it what you will. I have been writing a mini essay of sorts on the topic, which I began expressing parts of in this post about open source hardware and home fabrication.

Where do I start? Let’s start with the fact Peru had a large earthquake on the 16th August near the cities of Chincha (The Afroperuvian epicentre), Ica and Pisco. These places are approximately 140kms south of Lima, where my wife is from (and her family are living).

I subscribe to Treehugger RSS feeds and the same day as the Earthquake comes a post with Vinay Gupta, the brainchild of the Hexayurt project, an open source disaster relief shelter, to put it simply. I suggest Treehugger subscribers pitch in some funds to send Vinay to Peru to kick something off and Vinay responded with this post, to which I also responded. When Vinay was discussing the Hexayurt, he mentioned “wood gas fire” as seen in this clip

I happen to work with a guy who is right into his steam engines.. now, I don’t really discuss my “treehugger” leanings with him, as they burn coal in those things but the above wood burner did lead me to think about how a closed system steam engine might possibly be able to fit with the goals of yet another open source project called worldbike, which is a group of bicycle designers who make open source bicycle designs to build or modify an existing bike to be more suited to our brothers and sisters in Africa.

I would like to think that the goals of Open Source Green Vehicle are somewhat similar, in brief – a group of people who come together in like mindedness to share some of their skills for the betterment of humanity. People may not realise just how much Google has it’s hand in all this. Google is almost like the glue of all knowledge. Google has shown us to realise how powerful open business models can be and helps reward people for sharing their knowledge. Staying with the our OSGV project, for the moment. I came across this group of videos on youtube (now owned by google) that discusses in depth two big interests of mine. How to convert an engine to hydrogen and use it for both automotive use and / or when used in housing (whether emergency or not) it can provide a source of electricity, heating and clean water. Now, with the amount of instructional video content on youtube, it’s really a knowledge sharing platform just as powerful (if not more so) than Sourceforge or Wikipedia.

Vinay also linked to this Microcredit system being used in India, another area which I am interested in and I am not bullshitting about this, you can view my Amazon wishlist on this. I am currently on three weeks holiday. The first week (right now) I am in Canberra, the Capital of Australia doing my Vmware Virtual Infrastructure course. The following two weeks, I will be starting to code the beginnings of a Microfinance System that I will be writing in Django. I have been really working hard teaching myself Python (and Spanish) using my own modified version of the “Seinfieldian” system.

Now, it’s also worth mentioning that through Vinay’s posts, I ended up finding “The Open Business Guide” and Open Source Ecology that Vinay offered up to me, which I haven’t yet had time to have a good read of as yet.

I could write a lot longer of the “bizaare” sequence of events but I have some reading to do.

Richard Stallman may yet become the modern day Marx, the way things are going..

The simplest example of open source that I use as an analogy is

“imagine eating in a restaurant where you loved the food and the chef was willing to share the whole recipe with you – you may go home and make it yourself but if he is a good chef, you will no doubt return for ‘his way’ of preparing the dish” – this is the nature of open source, to give something to humanity and yet you lose nothing.

Powered by ScribeFire.