Benjamin Meyer's Blog

parallelizing sequential work in amdahl's law

2012-01-21T04:36:00.000-05:00

The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program.

When a sequential fraction of the program is the act of splitting up the data you can remove this sequential work by parallelizing the splitting and pseudo randomly picking split points, later tossing out duplicates in the join step.In this senario you trade the requirement of extra computing power for faster results.

Using collaborative diffusion rather than path finding for the Google AI ant challenge.

2011-11-02T20:15:00.001-04:00

For the 2011 Google Ants AI Challenge rather than doing the typical solution of choosing direction for each ant based upon the shortest path to some goal I used a diffusion based approach which was simpler, faster to code and resulted in some nice emergent behavior with very little work.

A while back I read the paper Collaborative Diffusion: Programming Antiobjects. The 2011 Ants AI Challenge seemed like a perfect test for it. Rather than having each ant, hill, water etc be its own object, the map is the only object. The map computes the diffusion values for agents (such as food and searching) on each square and then each decide where to have each ant go not based upon any path finding algorithm, but simply based upon the diffusion values surrounding the square.

Each square on the board has several agents or diffused values (food, explore, hill, etc). On every turn the program would loop through the map setting or diffusing all of the squares and then loop through the ants using the diffused values at the ant's square and some very basic logic to decide which way the ant should move.

When running it against the test bots even the simplest version would get neat emergent behavior such as in a maze when two ants follow each other down a hall when they reach a fork they will naturally go separate ways or when there are five bad guys against the one good guy the diffusion value wont say to attack until you have six guys.

The two simplest things ants want to do are get food and explore. To make the ants go after food we first have to diffuse the food value and then have the ants go to the neighboring square with the highest FOOD value.

A snippet of slightly modified code from my diffusion function which is run on each square on the map.

void Map::diffusion(int r, int c){
std::vector goalsToDiffuse;

Square *square = &grid[r][c];
// water blocks everything
if (square->isWater) {
square->agents[EXPLORE] = 0;
square->agents[FOOD] = 0;
return;
}

// FOOD
if (square->isFood) {
square->agents[FOOD] = INT_MAX;
} else {
goalsToDiffuse.push_back(FOOD);
}

// EXPLORE
if (!square->isVisible) {
square->agents[EXPLORE] = INT_MAX - ((200 - square->lastSeen) * 300);
} else {
goalsToDiffuse.push_back(EXPLORE);
}

...

foreach (goal, goalsToDiffuse) {

double up = upSquare->agents[goal];
...
square->agents[goal] = 0.25 * (up + down + left + right);
}

To diffuse food when a square is water it should set the food diffused value to 0 so ants never want to go into the water when looking for food, otherwise when a square has food set the food diffused value to a large value. This large value is the final goal for the ants. Lastly if we are not water nor food we simply diffused value to the diffusion value of the neighboring squares.

To diffuse exploration I include in the information of when we last saw the square into the diffusion value so my ants always go to the part of the map that has been least explored and where they can potentially find hills. This prevents them from bouncing up and down, but always moving forward.

After diffusing you want to output the moves. This is where the real logic comes into play. My first version was similar to the following pseudo code:

foreach(ant, myants) {
food = valueOfFoodAt(ant);
if (food != 0)
goal = FOOD;
else
goal = EXPLORE;
outputMove(ant, goal);
}

outputMove will chose the neighboring square (N, S, E, W) with the highest goal value. In effect the ants will move to the food/explore/hill/etc via the shortest route.

The results are are fun to watch and with just the above code the ants will explore maps of any shape be it mazes or open lands eating up food and and as more and more ants are born they will naturally separate across the board waiting for new food to appear always moving to the squares that it saw least recently. Adding a HILL to the above diffusion would be similar to FOOD, and the logic for the ant movement is as simple as giving it a higher precedence than FOOD. And once the diffused HILL value reaches the ants again the map doesn't matter be it a maze or open land, the ants will march to the hill via the shortest route.

The diffusion approach to the ant problem was simple to code (no shortest path algorithms) and I was able to get the first version up and working in about an hour. The runtime is O(rows*cols) (notice it isn't dependent upon the number of ants), the cpu time is extremely small compared to other solutions, and the memory overhead is pretty much non-existant compared to other solutions.

Given both the ease at which this is implemented and the small number of lines it would be nice if a basic diffusion based solution with goals for food, explore and hill were included with the stock python bots that the aichallenge package includes. It would be both a great starting point for making more advanced diffusion bots and as a stronger test bot, but one that still used very little cpu.

Anti-objects are a neat concept that I don't hear too much about and something I wish I had explored sooner. If I was introducing someone to programming the ants problem combined with a diffusion solution would definitely be a small effort, huge reward way to go. The simplicity of the algorithm combined with the low memory and cpu overhead also was an aspect that I had not looked at before and could probably be utilized to make some great cheap behavior for AI in games, demos and other places.

Qt on Blackberry

2011-10-18T18:58:00.001-04:00

Today it was announced that Qt will be included in the BlackBerry native SDK so you can put your Qt apps on BlackBerry devices.

http://blackberry.github.com/ndk/components.html

The coming IT revolution

2011-10-16T23:37:00.001-04:00

Corporate IT departments will soon fail to exists as we know them today. Any useful service that can exist on the internet will eventually be moved there. IT departments that cling to systems that they maintain and will only be harming their company. Companies that provide services on the internet that IT departments provide today will grow and prosper.

More than a decade ago when the KDE project was born a small group of individuals setup the necessary resources so the project could succeed. The most important bit was the cvs server, where the source code actually lived. But on top of that were countless other services such as a bug tracker, web servers, mailinglist server and more. Part of the success of KDE was these services. They were not just for one application, but they were setup in such as a way that if a young energetic developer wanted to contribute a new library or application it could be easily assimilated without the need for him to setup all of the services himself. In trade for following the KDE way the application got fame, contributions and a host of services.

But there will never be another project like KDE and what allowed KDE to flourish and grow in the early days was one of the reasons I stopped contributed to it¹. For the past half a decade there have been more and more online services. Today that same young developer wouldn't add his application to the KDE ecosystem when he could use any number of online services such as GitHub, Gitorious, SourceForce, GoogleCode, and the various Google apps.

A few weeks ago Jason over at 37signals wrote a blog about sneaking into the fortune 500. It has been rattling around in my head ever sense. In 2010 I created a GitHub clone called GitHaven for corporations intranet, but while there was interest and money in GitHub:Fi it is pennies compared to GitHub.com. Chatting with a startup recently we were discussing how they were using GitHub, Dropbox and probably would use GoogleApps for email. Just like services for open source projects there was now general services that startups have been taking advantage of and while they were kicking and screaming corporations were starting to use online tools too.

Five years ago there was a mad dash to create an online version of office. While it didn't take off a quickly as so many predicted (it is still a success) the corporate version of GMail did what no one had been able to do for more than a decade, replace Exchange as the choice for mail server. Once you can replace Exchange you can replace anything.

We are at the mist of a land grab for some extremely valuable property. Anything generic that was once done by corporate IT departments will be moved to the web. This means that IT departments will be shortly be going under a revolution and those companies that provide IT services could be very profitable as they scale services across many companies.

1 When I started developing my web browser Arora in 2007 I put my code on GitHub and used GoogleCode for the bugtracker, wiki (GitHub didn't have either of these back then), website, and Google groups for the mailinglist. There was no value in joining KDE's infrastructure and I saw being forced to use kdelibs as a downside as my application could no longer be distributed on Windows or OSX, the KDE release cycle was too long for my application, and to top it off while KDE had been one of the first groups to switch to SVN from CVS earlier in the decade it was not making the leap to Git in any hurried pace (they have sense switched).

GitHaven

2011-10-09T22:13:00.000-04:00

I am pleased to finally announce GitHaven which is a web interface for managing Git repositories.

GitHaven was created out of the need to be installed behind a firewall inside a corporate intranet or for personal use at home. GitHaven is packaged as a Debian package so it is extremely easy to install and keep updated.

Besides being open source the one feature that GitHaven has the GitHub doesn't is the ability to add top level tabs. So if you want a tab called "Wiki" and it points to your wikipedia entry the Admin can add it. This allows you to easily incorporate with other tools (mostly bugtrackers which are often jira, bugzilla or other)

While I still use it for my own personal use I am no longer developing GitHaven so it has been released under the AGPL in the hope that it can be useful for others.

Programming tool: The whiteboard marker

2011-04-11T08:00:00.003-04:00

All through my programming career I have had a whiteboard, but beyond simply making sure I had one I have never thought much more about it. Recently I picked up a set of whiteboard markers that surprisingly made me more productive.

Back in college me and a friend went to Home Depot and picked up a 4x8 sheet of bathroom melamine for $16. This is like normal whiteboard melamine, only slightly thiner and sold to a different demographic that buys it by the sheet and not by the inch. Cut in half we for a grant total for $8 we each had a very large whiteboard. To go with the cheap whiteboard I would often grab the cheapest set of markers at the store (usually a set of four Expo's) and think nothing more about it.

Last week when looking for some more markers I stumbled across this set of low odor ultra fine whiteboard markers with eraser for a grand total of $8.99.
A number of very interesting selling points that convinced me to buy it. Here is a shot of one of the new markers next to one of the normal fat expo markers.

Over the years I have worked at a number of different companies all around the world, but they all had the same big Expo markers. I have used thin expo markers, but these were usually the cheapest ones bundled with small whiteboard that never worked very well. Between those experiences I never questioned if there was something better out there beyond the fat Expo markers. That is until now and after using these new markers for only a few days I know I can never go back.

The four features that sold me on these markers:

Eraser On the cap is a small eraser so when you make a small mistake you just flip the pen over and erase it. No more hunting for the eraser as it can causes you to take your eyes off of the code and loose your train of thought. The little eraser can also cleanly erase small lines, letters and words while using the big eraser often results in a mess.

Low Oder A small thing sure, but after any long time of thinking and drawing on a whiteboard a headache is the last thing you want.

Grip Another small thing, but having a rubber grip is way more comfortable when writing on a board for more than a few minutes (let alone multiple hours).

Fine tip With a fine tip you can simply fit more "stuff" on the whiteboard. Whether your whiteboard is 8x4 or as many offices have, only 2x3, space really matters. With a chiseled tip you end up writing in a larger font size in order to make it more legible (or is it the same size but bolded?). This results in taking up more vertical and horizontal space.

When standing up and writing on a board there is a vertical zone that a person can write best in. Outside of that zone your arm and wrist and not optimally placed resulting in sloppy handwriting or crouching down.* Using the fine tip I use less vertical space per line and can write more in this legible zone.

I did a test this morning writing out a binary search function using the fat (blue) and thin (blue) markers (click on the image to zoom). I tried to ignore the other side and just wrote whatever was comfortable. For horizontal comparison there is a red mark on the left hand side which shows where the width of the blue function would end. When viewing the image also notice how some of the loops in the e's written with the fat tip become a blob v.s. with the thin tip. Reading code written with the thin tip is just easier.

When talking about programmer tools usually compilers, debuggers and editors are the first things that come to mind. Asking some more you might hear about multiple monitors and chairs, but I have never heard whiteboard markers listed. At $8.99 this is one of the cheapest ways I have improved my productivity in a long time.

*Or mount your whiteboard on tracks so you can raise/lower it.

Tablet TV Games

2011-03-17T18:44:00.002-04:00

1) A lot of people have smart phone and some have tablets.
2) A lot of people today can browse the web on their TV. This is done via an internet enabled TV, an attached box like a Wii, XBox, boxee, or something else.

Both of the above will only be increasing as time goes by. What major unique situations does this present that has not been seen before and how can it be taken advantage of?

1) Each user has a screen that can be private while still being in the same room.
2) Not locked into only 2 or 4 players on split screen, how about 20 or 400 people?

Some other minor attributes of this type of setup:
- There is a common screen + sound device
- In person communication / same room BUT users can go to another room to discuss if needed and still have their controllers / private screens
- System can also involve some physical objects due to the fact that everyone is in the same house.

Some initial (mostly bad) ideas:

- Poker
- Football simulator (other person can't see what play you pick)
- Pictionary (pass around a device which people draw on which is shown on the screen)
- Each device is a first person car racing game, TV shows the track / location
- WW2 B52 Bomber simulator. Each tablet is a first person perspective of one of the jobs in the bomber. The TV would contain the radar etc requiring users look at the TV goes with the feel of the plane and the required real time communication works well.
- Multiplayer airport landing game (each ipad would control a set of planes and or airports and the TV would be a zoomed out view or the multiple airports)
- Any coop games where getting more people helps (chaos equals fun)
- Some turn based board games
- Presentation "software" (at is core tv + 1 controller)
- A Kitchen game where orders are shown on the tv and ach person is in charge of preparing part of orders. Lots of talking would hopefully result
- Train game where each person controls part of a subway/train system
- Snow plows. TV shows a city and each person drives a plow communicating where they are going to go to clear the snow.

Downsides:
- Sitting on a couch the tv is pretty far away so resolution isn't as good as a board to a board game one foot away
- controller screens are down in your lap, tv is up high, makes real time games a bit hard as they are on different eye levels.

Overall jumbled thoughts on this topic:

1) The distance to the TV from the users limits the usefulness of any game

Given the distance from the couch to the TV I wonder if for many games what is shown on the monitor could just as well be shown on the tables/smartphones. A monitor on the dining room table two feet away can provide a much bigger map than a TV 6 feet away. The monitor also solves the browser compatibility problem. The monitor could also display from an ipad etc. Also given that a TV doesn't have a touch screen the user has to some how interact with what is shown on the TV which results in some representation of the screen data also being shown on their device. More thought needed....

2) Private displays are the key, but when each user has a rich tablet display why not just skip the whole TV requirement and just make a bunch of co-op games for the ipad?

3) Barriers to entry on the TV
While TV's might have browsers it isn't the latest and greatest chrome. Much more likely to find an older opera etc and slow javascript engines. Requiring everyone log into a site tvgames.com adds one more chance for people to not bother and do something else. Compared to ipad's that find each other with no user work required.

4) Tried before?
Back when the wii first got a browser a number of Wii browser games were in the news, why didn't that take off? Maybe all of those DVD games just get bought, but never played? Some Wii game sites on the first page of google show less then 400K hits in the past several years combined... ouch

Update: Looks like the Wii2 scheduled for 2012 will have some sort of touch screen. No doubt it will be able to take advantage of the above items.

Book Review: "Making Software : What Really Works, and Why We Believe It"

2011-02-21T18:27:00.001-05:00

Last week I received Making Software : What Really Works, and Why We Believe It. From the first moment I heard about the book I couldn't get the idea out of my head. Throughout the years I have read many books and many more blogs discussing how X is better than Y, but here was a book that was going to go through listing what we really know with studies to back it up. Rather than just claiming office cubes are bad or pair programming is good the essays present research, evidence and credibility to what they are saying. I couldn't wait to read it.

Just a few of the many questions that it tries to answer:

How much time should you spend on a code review in one sitting?
Is there a limit to the number of LOC you can accurately review?
How much better/faster is pair programming?
Does using design patterns make software better?
Does test-driven development work as well as they say?
How much do languages matter?
What matters more: How far apart people are geographically, or how far apart they are in the org chart?
Can code metrics predict the number of bugs in a piece of software?
Which is better: offices or cubes?
Does code coverage predict the number of bugs that will be later found?
What is right/wrong with our bug tracking systems today?
Why are graduates so lost in their first job?
Many more...

Short trailer for "Making Software" by one of the two editors Greg Wilson:

The book is broken out into two somewhat different sections. The first eight chapters discuss "The Quest for Convincing Evidence". It presents the how and what to collect and lists some existing data that is available. It also talks about when to know your are convinced, how to get your ideas reviewed etc. I was sold on the book for the chapters 9-30 which each cover specific ideas/theories so found this first part a little annoying to wade through as I was unprepared, but it gave the book a solid foundation and set your expectations of what the rest of the book was made up of.

Starting at chapter 9 it is broken out into a number of different essay's each written by a different author. Go skim the Table of Contents for a complete list of chapters. I can bet you that there is at least one chapter you would be very interested in reading this very moment.

Disclaimer: Before getting into some of the results many (if not all) of the chapters had nice big disclaimers about all of the possible issues and places their data gathering or testing could have failed. Any of the below statements are my take away from reading the book and things I am going to do differently. And if anything the book very much encourages (and teaches) you how to generate your own data review/publish your results.

Chapter 11 on Conway's Corollary really drove home the point that software is social.

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

If you have a large number of modules and you wanted to know which one was the most buggy the best predictor is what module has been modified by the most number of groups in the company. Code coverage, testing, edits, code churn, dependencies, number of pre-release bugs etc all give lower precision. I would love to see a script that took in a git repo and outputted a org chart. Then you can compare it to the real org chart to discover where things are probably a complete mess. Communication structor, developers leaving, number of engineers working on a project are all discussed. But it isn't all bad. The chapter also discusses how you can exploit this to leverage the corollary.

Chapter 17 discussed pair programming which is something I have been debating trying to do and now I know I will. In one example cited while one person would take 10 hours to complete a task two people pair programming only 5 hours and 45 minutes, but the expected 10 hours. Not only does it take about half as long, but the resulting software was of higher quality and you get cross training in the process. They also discuss one v.s. two keyboards, experience levels and more.

Chapter 19 provides you with the evidence to convince your VP that you should go to offices, war rooms or some combination and get rid of your cubes. Don't just say you will be more productive, show him the chapter so he/she can read and decide for themselves.

Chapter 24's report on bug trackers was more interesting than I thought it might be. The biggest 'discovery' (in my opinion) found was that marking a bug as 'Duplicate' in a bug tracker causes data to be lost. The duplicate bug's often contained rich information that would have helped a programmer solve a task quicker, but by marking it as duplicate developers often don't see it. I would love to see bugzilla and other bug trackers get a Merge feature rather than Duplicate so reports are not lost.

Chapter 26 discusses how the problem many employers have with hiring collage graduates where they are not able to utilize them as quickly as they would hope. I have seen various thing tried, but wasn't sure what worked best. Microsoft tackled this by following around some new developers and wrote down every single thing they did. Like a UI review where you can only watch a user and can't say anything as they fumble around I was frustrated reading the reports of what the new developers did. They would get assigned to projects (not very well documented) working in groups (v.s. on their own like in school) using internal tools that they have never heard of. It is no surprise it takes some time to get up to speed and for the most part they were just really confused by a lot of things. Close mentoring and giving the mentor actual time to mentor is very important. The chapter includes more tips on changes companies can make.

Chapter 27 discusses how to look at your own code and evaluate it. After investigating some of my own open source projects it became clear that I need to do a better job of being able to distinguish what a change does, but even with just some basic dumb analysis on the repository it showed interesting results.

Chapter 28 "Copy-Paste as a Principled Engineering Tool" was very surprising to read. I have always been an advocate of the DRY principle, but never thought about it as much as they did. Here they take apart DRY and even break down and can classify DRY violations into different types and end up arguing that while many are bad a select few are good/necessary.

Chapter 29 is "How Usable Are Your APIs?" It tells the story of designing an API at Microsoft and just how very important it is and tips and tricks you can do. All throughout reading the chapter I kept thinking that if only the author had read The little manual of API design [pdf] it would have saved them a lot of work. If you haven't read the little pdf I highly recommend taking the next twenty minutes and going through it.

The book is filled with more more discussion points. Each chapter can be broken down and discussed in rich detail. There is so much in this book it is hard to give any sort of summary that does it justice. Each section comes backed by a ton of references so you can look up the data and not just take the authors word. The book reminds me of Diffusion Of Innovations which also reads a bit like a text book but is overflowing with information where whole books can be written on just one small part. Developers, Managers, Testers, Human Resource, the reach of those who can get value out of the book is large. There were a number of sections I held beliefs on and didn't think I would learn anything, other sections I was very curious to know what it might say and others that completely surprised me. This book now sits on my shelf next to Mythical Man Month (dare I say replaces it?). If you only read one technical book this year make it this one.

Edit: Some more links including a webinar discussion on third-bit.com.

Playboy for my birthday

2011-01-17T18:33:00.008-05:00

This year for my birthday one of the presents from my wife was my first issue of Playboy. Not just any issue, but the November 1972 issue which contains the Lenna Sjööblom centerfold. A small portion of this centerfold is a very well known image in computer science. In 1973 Alexander Sawchuk tired of test lines and tv patterns grabbed a playboy from someone walking by and scanned the top 5.12 inches (the image seen on the right) to be used on a colleagues paper. This scan went on to be used in countless times in all sorts of image related research such as compression, watermarking, line detection and more. It is one of those images I have come across many times when reading papers.

Last year when watching Pawn Stars someone dropped off some boxes of Playboy's only to find out most of them were not worth anything and how the only issue that people want is #1. I made a comment to Jen about how the only issue I would want would be the one with the Lenna, explained its history and how it must be worth a lot because everyone knows about the Lenna image and reads image papers ... right? Jen secretly wrote that down and later on was able to find the issue, acquired it and stashed it away until my birthday. That issue was Playboys' best selling issue ever so it isn't super rare. While it might not be the most valuable (yet!) it is one cool birthday present to get from your wife.

Evolving "Hello World!"

2010-12-14T00:14:00.005-05:00

puremango has put together a nice little writeup on using genetic algorithms to generate the classic "Hello World!" He has an attached webpage that lets you play with the parameters and generate a nice graph.

A few years ago I put together a generic genetic javascript algorithm webpage. While it doesn't have the nice writeup or the cool graph (only a basic one) the cool feature was that you could modify the input box to put in your problem. Taking a few minutes here is the hello world! version that you can put in the text box and run to have it discover hello world.

function salesman() {
this.target = "hello world!";

this.fitness = function(chromosome) {
var f = 0; // start at 0; the best fitness
for(var i=0, c=this.target.length; i f -= Math.abs(this.target.charCodeAt(i) - chromosome[i]);
}
return Math.abs(f);
};

// the size of values that should be passed to fitness
this.numberOfArgs = function() { return this.target.length; };

// the max value needed for the arguments
this.maxArg = function() { return 128 }; // ascii

// convert the current chromosome value which can have a maxValue
// into something fitness can use.
this.getArg = function(value, maxValue) {
return value;
};

// Paint the solution onto bestimage
this.paint = function(values) {
var canvas = document.getElementById('bestimage');
if (canvas.getContext){
var canvasContext = canvas.getContext('2d');
canvasContext.clearRect(0, 0, canvas.width, canvas.height);

canvasContext.font = "italic 200 12px/2 Unknown Font, sans-serif";
canvasContext.strokeStyle = "blue";
for (var i = 0; i < values.length; ++i){
canvasContext.strokeText(String.fromCharCode(values[i]), i*10, 10);
}
}
}
} salesman;

Git SSD Speed comparison

2010-12-13T13:27:00.002-05:00

SSD's provide significant improvement in disk IO. How well does that translate over when using Git? A lot.

After watching prices drop all year long during black friday I picked up a OCZ-Vertex2 120G SSD. I installed it in my current MacBook Pro (2.66 GHz i7 with and 4GB of ram) using a MCE OptiBay to replacing the DVD drive so I can have both a HD and a SSD installed at the same time.

All tests were run in Snow Leopard using the Git 1.7.3.2 pre-built binary package. The HD is a 500G Toshiba MK5055GSXF with a Negotiated Link Speed of only 1.5 Gigabit.

Test 1
Creating a new repo, populating it with the contents of linux-2.6.36.1, add commit, create a branch and run git status.

The git add took around ten minutes to run on the hard drive compared to the 40 seconds for the ssd. Committing was also 1/3 of the time taking only 20 seconds v.s. over a minute.

The ten minutes to perform the git add operation on the hd made me think something must be broke and it just seems beyond slow. Turning on/off spotlight and everything I can think of in OS X makes no real difference. To further eliminate variables I mounted an old MacBook HD (Toshiba MK8034GSX 80GB 5400 RPM) I had lying around using a USB connection to my mac and found the following results:

git add - 6m30.422s
git commit - 1m47.226s

In conclusion on OS X slow 5400rpm drives produce slow results and ssd are really that much faster. Moving on...

Zooming in on the graph we can see the improvements to the other commands. The biggest win for me is that git status, a command I run all of the time (I have aliased to gs) runs in half of the time.

Test 2
Cloning a large repository from a local network was the next test. I was hoping to see improved performance on verification and it was faster, but compared to everything run on the server and transfer speeds the seven seconds saved wasn't that much compared to the seven minutes the server (a slow arm based box) was preparing the repo.

Test 3

Removing the network here is a clone of the Qt repository from the hard drive to the ssd and hard drive to hard drive. Followed by some simple walking.

Walking was faster on the SSD, but as soon as the data got into ram it was equally fast for them both.

Summary
As expected Git runs faster when its data is on an SSD. For operations like git branch which just create a file with a sha in it the performance isn't much different, but when running an io intensive operation like git add you can see that it is extremely worth it. Git status was probably the big winner by getting a speed improvement of around 50%.

Averaged data:


Creating a new repo and populating it with the contents of linux-2.6.36.1

         HD          SSD
init      0m0.011s    0m0.010s
add       11m2.903s   0m40.041s
status    0m2.323s    0m1.138s
commit    1m4.097s    0m20.410s
status    0m2.027s    0m1.532s
branch    0m0.039s    0m0.009s
checkout  0m1.465s    0m1.473s
status    0m2.414s    0m1.073s

clone qt-snapshots from local network
clone2    7m21.066s   7m13.310s
rev-lis   0m0.082s    0m0.077s

cloning qt (32611 commits) from hd with --no-hardlinks
clone     0m36.770s   0m13.050s
log       0m1.042s    0m0.977s
rev-list  0m1.697s    0m0.767s

For those on Linux checkout some linux ssd tweaks.

Our first House

2010-10-29T22:11:00.004-04:00

After years of saving, months spent looking at more than fifty houses and weeks of paperwork, today we closed on our very first house.

We are happy, excited and can't wait to move in.

GitTogether10

2010-10-28T19:36:00.000-04:00

This week I went to California this week and and attended GitTogether10 that was hosted at Google. It was a small conference all about Git. From hearing how companies are using it to discussing how to make Git better the discussions were varied. I got to meet a bunch of very cool developers and hear some very neat work that is going on such as bup a backup tool using git that can continuously backup huge files such as vm images without taking up huge quantities of space. There was a lot of Gerrit users there so we ended up taking our group photo in front of the Android statues:

As a thank you present Google gave out Nexus One phones and a little Android mini collectable.

The future of the MacBook is the MacBook Air

2010-10-21T23:58:00.001-04:00

The new MacBook Air is not something to compare to netbooks, but a roadmap of where the MacBook is going with its improved performance and reduced size and parts.

Obsolescence
Back in 1998 Apple removed the floppy from the iMac G3. There is no argument that it was obsolete technology. Small, easily damaged with better replacements like the zip drive, cd-r, usb sticks, and networking. For those who still had floppy disks you could get a usb drive. The iMac G3 led the way and the rest of Apple products followed suit.

One of the primary data replacements of floppy was the "cd". The cd drive, cd burner, dvd drive, and finally dvd burner while useful is less useful than it used to be. Most backups don't fit on dvd drive. Software and games are usually downloaded and not bought in a package these days. CD's are no longer bought and ripped, but usually obtained also online. Even movies are transitioning to streaming services. Even creating home dvd movies has been depreciated in favor of posting online. The big final reason was the OS install disk, but that is now just a usb stick. Just how back in 1998 we still had a pack of floppies disks on the shelf we have dvd's lying around. But we can get one external drive to share among the computers as we transition. I myself plan on pulling out mine from my MacBook Pro and putting it into an external case with a optibay. Replaced with the internet and usb sticks disk drives are dead.

As for ethernet, 802.11n is around 130Mbps (real world) which is decent for most everything. Other than transferring GB's of data WiFi is good enough. Backups are the biggest thing most users will transfer, but even that is done overnight (over WiFi!) and in the background. After the disk drive the ethernet port is pretty tall so removing it allows for thiner laptops. At the end of the day, like the floppy if you really need an ethernet port you can get a usb version and just like the Firewire port it will get the axe on MacBook models.

IO
The last ten years the speed of the cpu on the computer has had less and less of an effect on the performance than one would think. First was the race to 4GB of memory, followed by graphics and now io speed.

The most dramatic improvement someone can make to their computer these days is to switch to a solid state hard drive. As the size of flash has continuously increased while the price has dropped it becomes more and more attractive for everyone from desktops to server farms. There is a lot of research and experimentation in this area with leaps in performance coming out all of the time. The next several years will be very interesting to watch as flash grows. While it seems mature flash is still new enough to warrant whole blog posts by users who just upgraded extending the virtues. From the lower power requirements, the lack of any noise, fitting in different shapes, and of course mind boggling speed, ssd drives are the future. Apple has offered flash as an option for a while now on some of its computers, but with the new Air it takes the option away and makes it the default. As prices continue to drop and sizes rise ssd will become less and less the option and more the default (or only) option.

tl;dr
It would be no surprise if in the next refresh or two the macbook moves to ssd and drops the super drive and ethernet port. The MacBook will be thiner, have a longer battery and a faster user experience.

As the tagline says:

The new MacBook Air. The next generation of MacBooks

Managing project, user, and global git hooks

2010-06-19T15:15:00.003-04:00

I recently put up a new tool on GitHub called git-hooks. This is a tool to manage project, user and global Git hooks for multiple git repositories.

Hooks can be very powerful and useful (as I have previously written about in my git hooks blog post). Some common hooks that I always want running no matter what git repository I am commiting to include:

- Spell check the commit message.
- Verify that any new files contain a copyright with the current year in it.

Every project also has specific hooks that I run such as:

- Verify that the project still builds
- Verify that autotests matching the modified files still pass with no errors.
- Pre-populate the commit message with the 'standard' format.
- Verify that any new code follows the 'standard' coding style.

And I have some hooks that are very specific to just me such as:

- Don't allow a push to a remote repository after 1am in case I break something and will be asleep.
- Don't let me commit between 9-5 for projects in ~/personal/ as I shouldn't be working on them during work hours.

In the past I have just been copying these from one git repository to another, making symlinks or other nasty hacks. I finally sat down and wrote a tool to manage these different hooks.

After running 'git hooks --install' in a repository rather then using what is in .git/hooks the following locations will be checked for hooks:

- User hooks that are installed in ~/.git_hooks/
- Project hooks that are installed in the git_hooks directory inside of each project.
- Global hooks that are specified with the hooks.global configuration.

If you have many hooks you don't have to write a wrapper to call them all, they all simply go inside of a directory (such as git_hooks/pre-commit/bsd.sh).

git-hooks/contrib also includes various simple hooks I are useful that you might like to use. git-hooks is released under the BSD license.

What makes a fun Mario level?

2010-04-26T01:00:00.007-04:00

This years 2010 Mario AI Championship includes a level generation contest. Having been interested in creating a platform game for several years I have written up my notes on Mario levels for those who are working on an entry. Disclaimer: I have never worked on a shipping platform game so this is simply what I might try to achieve if I was making a level generator.

Basic Rules:

While objects could be places randomly, good levels that are generated should at the bare minimum follow the basic Mario universe rules such as:

A pipe flower must come out a pipe, not the ground
Fish are only found in the water and other bad guy should only appear in their respective level themes. i.e. Bowser shouldn't appear in level 1
Mario must be able to reach rewards such as coins or boxes without dying.
All platforms graphics have left, middle and right images. They should all be used.

These might seem overly obvious, but I have messed around with enough home grown Mario games to see that it needs to be stated.

Average Rules

On any given level you will find the majority of the following to be true:

A maximum of seven bad guys on the screen at once by default, lower is better. If you play Mario 1 you will find that there are often only one bad guy on the screen at at time. The more bad guys, the more harder it is and the less fun. Two or three seems to be the magical amount.
Mario must be able to reach all platforms. A platform that mario doesn't think he can reach should only be there to lead to a secret and can in fact be reached via a hidden block, flying or other means. Everything should have a purpose.
There are between 50 and 100 coins in a level. Play a level twice and get an extra life.
A Mushroom box at the start of the level.
One save point midway through the level.
A Mushroom box after the save point.
The final bit of the level is always the same within a variation. In many Mario games you had one bad guy, a hill/tower/blocks. In Mario 3 you had one bad guy and the box to give you stuff.
A level is only X wide or high. While rooted in the technical limitation back in the day this size is what players have come to expect so while truly massive levels could be created the average level should fall into the expected size.

Overall level theme design

The level should choose a graphics theme such as desert, forest, castle and stick with the graphics and bad guys from that theme. Having a water level with bats doesn't make much sense.
At the start or end of a level there can be transitions from one theme to another such as leaving the desert and entering the forest.
Location: water/ground/air. Within each theme you can have water, ground or air levels. Walking on top of the castle in the air, water under a forest etc.
Does the level scroll vertically or horizontal? The generator should be able to create both types.

What is fun?

Getting to the meat of the contest is making levels that are fun to play. There are many different things that are fun and often times it is important to know what is not fun to know what is fun. Here are just a few:

Completing the level is fun. Unless it is world eight no level should be hard or take more then a few lives to get through. In fact many levels you have never seen before you should be able to get through on your first or second try. Dying is not fun. It is temping to make really hard or confusing levels, but don't. Let players run through the level and entice them back with the many ways levels can be replayed.
When you almost die (but don't) it is fun. Walking along, a block falls from the ceiling right in front of you making you jump. Blocks falling from the ceiling that you can miss if you just run is fun. The longer you can run without stopping is fun. Being scared, but not in danger is fun.
Finding a secret area is fun. Newer Mario games have about three in a level. The secrets shouldn't be completely hidden, there should be a hint such as a block in the middle of nowhere. Running along you could easily miss it, but if you went slowly you would notice it and after a little exploring you would find it. Finding secrets is a lot of fun because you want to show them off.
Choosing the right path is fun. When there are multiple platforms you could run across, one being fast and another being slow with lots of bad guys it is fun knowing that you choose the right path.
New bad guys are introduced in a learning progression. The first time you see a new bad guy he is all alone and you can approach it slowly and figure out how to kill it, only later having the bad guys come several at a time or unexpectedly.
Collecting is fun. The most basic collecting in Mario is the coins, other Mario games have you collect five big coins per level, MB3 had power ups that you could get at the end of the level.
Hidden rewards. Coin blocks, power ups, blue coins, one-ups. Nothing beats telling someone where the one-up is on 1-1.
It is fun to beating the level quickly. Maybe through a pipe warp or through a trick like running on the ceiling on Mario1 1-2. Players love finding ways around actually playing the game and love telling their friends.
Many levels tell a story along with having you run through the level. Ship levels have you arrive at the front of the ship and you progress past the masts and to the back. Level design can include nods to the story.
Many levels have a story with the bad guys. On a vertical level there could be a slow but constant stream of the bad guys. Near the end there can be two pipes with the bad guys coming out of them implying they all came from those pipes. Same can be said for many castle levels were you have fireballs coming from bowser who is far off. Having a story makes the level unique and interesting.
Highlight one game engine or sprite mechanic. While each level could simply combine the random elements it is much better to have each level be primarily about one thing. A sky level where you have platforms that move on rails. There might be a stationary platform or two, but mostly it is about the moving platforms. Ghost houses are another example where the levels are designed to explore the ghost bad guys and the different ways they can be presented to the player. Choose one aspect to highlight per level. Don't just create another bland Forest 4-2.

Procedurally world and game generation

The last section highlights what I think might be the Achilles heel of the contest. Even if I could generate a pretty good forest level playing 32 of them wouldn't be very fun as the same tricks and level designs would probably be repeated. It is a lot more fun to have several forest levels, several ghost houses, a coin ship, castles, ships, water levels, waterfalls, sand traps all spread out. Much more interesting then level generators would be the task of generating a world, such as the following task:

I just came from the forest world and am entering the desert world. Generate eight levels, two castles, one ghost house, one or two warps and include the following bad guys ....

Given those parameters a generator could introduce the bad-guys one by one knowing that in later levels you can easily beat them. It would be able to tie the levels together such as having one or two desert bad guys in the castle and have odd levels such as the coin ship which while not very fun by itself makes for a fun addition to a collection of levels for a world. World 1-1 should be very easy unlike world 8-4. A stand alone level generator doesn't know where it fits into the overall game or what levels came before or after it. It might just generate ten forest worlds in a row.

There is a random Mario game that you can play online. It generates a world of random levels for you to play, but I only played a few level because it became obvious that all of the levels were about the same and there was nothing new to play. Games that show you every trick in the book in one level are not very much fun later on.

"Fun" is a funny thing. Several years ago I created a random Quake level generator. Me and several friends knew all the Quake levels to well so one weekend I wrote a tool that would automatically generate a map, populate it with items, lights etc, compile it and distribute it to our boxes. Every ten minutes we got a new level no one had ever played before. Most levels were interesting and fun, but one day it generated a level that was a massive room with one wall in the middle. After spawning we immediately realized what we were in, grabbed a weapon and proceeded to run around this one wall. We spent several minutes chasing each other around this one wall laughing the whole time. While definitely the simplest and stupidest level it ever created it was a ton of fun, not something I would have ever set out to design. Following a set of rules for making a fun Quake level I would have never made what it had with my random rules.

I look forward to seeing the entires that are submitted. Writing up this has me itching to go back and play various Mario games to discover more fun aspects and to try out various ideas on my own level generator.

Image by KER

WebKit Meeting

2010-04-19T14:19:00.004-04:00

Last week I went to Apple for the WebKit Contributors Meeting. It was really great meeting so many people that I have only ever known through IRC and email. I had countless great discussions and even got some hacking done on some WebKit2 code. The WebKit community is very vibrant and I look forward to what will be added to WebKit in the future.

tagged names on flickr

Git Achievements

2010-03-15T10:00:00.011-04:00

This weekend I created a little application called git achievements. Similar to the XBox360 Achievements you can unlock all sorts of Achievements while you use git. It brings a bit of fun (and bragging) to those who use or who are learning git.
Almost all of the git commands have achievements of different levels (depending on number of times a command is used). For example you can become a "Apprentice Author" after only two commits, "Author" after sixteen and finally a "Master Author" after 128 commits. Various other Achievements can be unlocked just be performing an action such as adding a .gitignore file or setting your user name in git config. When you unlock an achievement you will get a message on your console showing the new achievement like so:

******************************************************************
                       Git Achievement Unlocked!

                               Caretaker
               Added a .gitignore file to a repository.
******************************************************************

Having collected all of these achievements you can set a git config flag and have it automatically publish your achievements to your github pages. The published page includes a list of all of your unlocked achievements, total points as well as a list of achievements that are still locked. I have my git achievements published so you can checkout the full list of achievements.

Installation is a snap, after forking and cloning the repository just add git-achievments to your path and alias git=git-achievements.

Creating this was a lot of fun. It was simple and a bit silly. After getting the idea I had something working in no time at all. It was fun coming up with the different names for the Achievements and as I was working on the project unlocking them. Originally I had only planned to create the output to the console, but it only took a few minutes to realize just how easy it was to publish to GitHub Pages which brought the achievements to a whole fun new level. Achievements in video games often causes the user to explore parts of the game that they might not have and often cause them to learning actions that they might otherwise ignore. I am not sure if git achievements is very useful as a learning tool, but it is fun to have installed and popup with a new achievement now and then. If there is a trick in Git that you think other people don't know about or should learn let me know (or send me a merge request) and i'll make a new achievement for it.

Photo by terren in Virginia under the cc license.

Anigma, a game made using only CSS3 Animations and Transitions

2010-02-16T12:00:00.014-05:00

This past week I have been playing around with the new css3 features. I wanted to learn more about the new animations and transitions API. The end results is a fun little puzzle game called Anigma.

Anigma is a simple game. To solve each level you must remove all the jewels. Jewels disappear from the board when they are touching another jewel of the same color. The jewels are stacked on the board so if there is nothing under them they will fall down. This can make the order in which you remove the jewels important on some levels. There are over 500 included levels of various difficulty. If you make it to the end there is a nice little end credits animation (nothing amazing).

All throughout the game you will find animations. From the jewels moving to the clock counting down your time left it is all handled by WebKit and not through a 3rd party javascript library. Using the HTML5 audio tag there are also a dozen 8bit style music tracks to enjoy. Absolutely zero flash is required to play this game, but you will need a WebKit browser (Arora, Chrome and Safari4 are all good). (Updated: Got it somewhat working with the nightly build of Firefox or the latest Opera beta both of which have initial support for transitions.)

I had a lot of fun making the game, but having spent a week on it I decided to wrap it up and release it. A few of the ideas that I have thought about adding to the engine to make interesting levels out of:

Elevators (I got this partially working in the js source, but ran out of time)
Teleporters
Ice blocks
Trap doors that would come down after you go under them.
Buttons that must have a block in it to activate something such as opening a door
Dumb waiter type elevator with two platforms and whichever is 'heavier' will go down and the other up.
Loose ceiling piece that falls and can crush a jewel
A better 'you finished the level animation'
Don't do anything and it will show a level being solved i.e. demo mode
Public High score board
Use JSON for the level format or something better then the text files.
Import the rest of the extra vexed levels
Generate a random level
'I give up button' that automatically solves the current level.
A snapshot state button so you don't have to restart all the way from the beginning.
Sort the levels based upon server logs for difficulty.

While it would be fun to implement many of those features it was really about showing some features of CSS3/HTML5 so having accomplished that I decided to stop. I have posted the source code up on github so you can feel free to fork the code and improve the game.

Hopefully you get a kick out of the game and get some neat ideas for what you can do with the new CSS properties. If you enjoy the game hit the 'e' key to bring up a level editor. If you create something good email it to me and i'll include it.

Founders At Work

2010-01-26T22:56:00.001-05:00

For my Birthday I got Jessica Livingston's "Founders At Work: Stories of Startups' Early Days". It was an enjoyable book and over the past two weeks I quickly went through it. The book is broken up into thirty-two different interviews with company founders. Most of the stories are about web companies, but there are a few hardware stories mixed in.

Covering so many different companies and individuals one coundn't help but compare them. You have companies like Apple that started with the brains behind it to make a really fantastic product and on the flip side you have Yahoo which looks like it was pure dumb luck. Yahoo was the biggest surprise for me. Started as a way to keep track of links for a dissertation the founders spent the first eight months manually entering in urls people would send them, sometimes eight hours a day the book says. They were in the right place at the right time and that was it. I found it extremely hard to read very deeply into any advice given from companies like Yahoo or HotOrNot which (another random luck story). But to match there are plenty of stories of companies who put in several years of hard work and it paid off. Luck is always a factor in a startup, sometimes it is tiny and sometimes it is a metric ton dragging you forward whether you know it or not.

The story of TiVo mentioned another company called ReplayTV which is no longer around. ReplayTV was TiVo's big competition when they were first growing. All of the stories in the book are mostly successful in the end (the company might have died, but the founder got to cash out). Discussing ReplayTV made me take note of the fact that there was no stories of failed companies. This is probably due to the fact that there are plenty of good stories without having to discuss failures and it is easier to sell a book about companies people have heard of, but no doubt the companies that failed had plenty of good lessons that they learned the hard way.

One story that resinated with me was the story of FireFox. While I lived through looking back I hadn't realized just how much they had going for them at the time. I.E. development team had been discontinued, Netscape only 5% of the users and a horrible interface and no change in sight. Out comes this little Firefox... Comparing that to Arora. Between new versions of I.E. that Microsoft is pushing, new versions of FireFox, Safari and then Chrome comes out. I am glad that I didn't try to really push Arora and kept it as just a QtWebKit browser that utilizes what Qt has to offer.

The book has a large quantity of material and at times I felt a few stories were just repeating earlier ones, but it was a fun read, I picked up a few things and while it doesn't get into a lot of technical detail there are lessons that you can pickup from the book such as: You will probably need to change your startup idea, it will be stressful, and patience and persistence pays off.

Hay Maze

2010-01-03T01:20:00.003-05:00

A week before Holloween Jen and I walked to Parlee's farm, a local farm just a few miles away where they had setup a seven acre hay maze. Going through a life size maze is one of those things I always wanted to do as a kid so when we discovered one nearby we went over on the next sunny weekend to try it out.

Walking through the maze took us almost an hour to get through. Even though the hay was only about four feet tall (at the highest) I couldn't see very far and wasn't able to follow were the trail went with my eyes because the hay would blend together. There are two bridges in the second half of the maze that you get to walk on top of and look out over the maze. Amusingly though when you got to the bridge and looked out you really couldn't solve the maze just by looking because the after twenty feet or so it is all just meshes together and you can't tell where the paths are.

This was very nice as I feared that I would just stand still look at the maze and then walk the correct path. This meant that I got to enjoy solving the maze with only the paths I have walked (I wasn't even really sure of the dimensions of the farm/hay fields). When you are solving randomly generated mazes on paper as a kid the only real rule was 'always take a right and you can't go wrong'. But with a life size hay maze you get all sorts of social clues that I quickly started to notice.

In a random maze generated by a computer one of the paths from the very first junction could lead you to the exit. When it is maze number 8 of 500 in a book that is ok, but for a hay maze that makes it not very fun. They want you to have fun and this knowledge can be exploited. Fun is spending a while in the maze, getting to go over each bridge and not wasting too much time on the wrong path. This particular maze was subdivided into sub sections. Each section was its own little maze with one correct path out to the next section. This way if you are lucky on one of the smaller mazes you might not be so lucky on the next (thus having more fun) and you can't spent too much time going the wrong way. Of course once you realize this you can exploit it by taking paths that would lead to where you think the next area would be and knowing you are going the right way if you have not reached a dead end in a while.

Another thing I noticed was that the creators of the maze made several sections that were designed for a human. In one junction that you enter you have four paths in front of you, but if you turn around you will find another path just to the right of where you entered the junction. Visually hidden from the entrance this path and separate from the other four it was clearly the correct path because it would be fun to discover it after walking down one of the wrong paths and returning.

Lastly given that there are people in the maze not only can you see them exploring areas ahead of you and behind you, but you can hear their joy and anguish. This quickly provides hints about what to do in the future. From how big a section is to which side has the right/wrong path. At times I couldn't help but laugh at the family who was stuck in the section behind us and the kids found dead end after dead end.

I went to the maze not really sure what I would expect. I thought it would be similar to every randomly created maze I have dealt with, but found something different and much more interesting. While actually getting through the maze was fun everything else about the maze was much more exhilarating. From ponding how one might make (and exploit) a "fun maze", being outside, people watching, and having fun with Jen made it a very memorable experience. If you ever get the opportunity to go through a life size maze I very much recommend it.

A Next Generation Desktop Web Browser

2009-08-15T23:44:00.002-04:00

The interface that web browsers provide today is one that is very similar to the first browsers that came out in the 90's. The back button, bookmarks, location bar are all very similar even though users use the web differently. On the smart phone there is a lot of experimentation of the interface, but the desktop hasn't changed much. Isn't it time desktop browsers got an overhaul?

History

Originally history was just a way to color links on a website so you knew if you had been to a link or no. To go with this history by default was stored only for a few days. These days browsers are designed to be able to store every single URL you ever visit. A history menu was a dramatic leap forward in usefulness for history and some browsers have begun to add more features to the history such as full history text search, screenshots, and more. While the history menu is nice it does have too many limitations. Is there a better way?

Bookmarks

A manually managed list of urls was nice back when the web was small, search engines were just being born, but the interface was created for adding urls, not managing them. Removing, sorting, filtering and searching is hard when by default it is shown in a menu. The Bookmark star solve so of these problems as it provides a way to easy remove bookmarks you no longer want. Some browser have begun to add features to the bookmarks such as the ability to tag, automatically updating, bookmarking rss and more. From adding, removing, managing, updating, syncing, meta searching everything about bookmarks is hard. How can a browser make bookmarks better?

Home Button

The home button is just a bookmark that happens to sit next to the location bar. Worse off it has a whole different way to be configured. Many browsers have already started removing this button by default. It is there because people expect it to be there. These days browsers run for weeks at a time, and have the ability to present a "new tab" page that shows the most common sites visited, arguably a much better home page.

Tabs

Tabs have problems. While there has been work in different design ideas on how to manage tons of tabs there are fundamental problem. We should not be putting time and research into how to manage hundreds of tabs, but into how can the browser help keep the user down to around only seven open sessions. Beyond the simple fact that users can't manager more then around seven of anything having hundreds of tabs means you could have hundreds of running JavaScript programs and you will run into system resource issues (not to bring up the fact that you have hundreds of dom's, images etc using up memory). As an experiment before opening a tab write down the reason for why you opened it. Very quickly you will discover the horrifying true problem.

Dialogs

Do you think asking a user to go down to seven tabs is radical? How about if if I asked them to only go to one? This is what is happening with popup dialogs. All dialogs related to a webpage should be part of that surface and not part of the browser. Already some browsers have begun to move some of the dialogs into the tab where it belongs. How can we move all of the dialogs into that tab?

"Private Browsing Mode"

Private Browsing mode as it is today if flawed. It was best highlighted when FireFox 3.5 users were horrified that there porn sites were showing up in the location bar search because they had bookmarked them in a "hidden" folder. When using Private Browsing it destroys much of the value that a browser can provide (history, bookmarks, autocompleting in the location bar, disk cache!) while not actually fulfilling the feature the user wants. Users don't care if they run into their porn, they just don't want anyone else to every see it. How can we provide what the user actually wants?

Visible URL

Before search engines you needed to put in url's a lot. It was only natural that this input field did double duty and automatically changed to the current url of the oage. Then we got a go button, autocomplete with history, the bookmark star, rss button, security warnings, a clear button, a site icon, search, and so on. The location bar turned in to the bastard widget where every feature gets put in it because it is always visible. While users still input url's with search engines and features like Top Sites this is much less so. Its primary function is to show the current url and removing that has to be done carefully because of security reasons, but already there are implementations out there such as the iPhone that show this can be successfully done. Ignoring all of the widgets dumped in the location bar the other purpose is to provide a crude command prompt into the browser for users. Is there a better way to display meta data, the current url and a way for users to input commands into the browser?

Menu Bar

Of course on OS X you will never remove the Menu Bar (unless you go full screen), but with no history, bookmarks, and dialogs, you don't end up with many commands. And with no location bar or bookmark toolbar it is temping to try to hide the menu bar to get screen real estate back. Making an Action button button like seen on browsers like Chrome shows that this is very doable.

Slightly better resolution than 320x480

There is a fair amount of research going on to make a browser UI for a screen that is 320x480, but what about a resolution that is 1920x1080?

Given that sites design to fix on a resolution of 1024x768 we can easily put two pages side by side on a monitor with that resolution. Given that that we don't have a menu bar, bookmarks (no bookmark bar) and are not showing the url or the site our interface (including window decorations) can be as simple as:

---------------
| META | META |
---------------
| | |
| LIST | SITE |
| VIEW | VIEW |
| | |
--------------

One of the primary reasons for multiple tabs is websites like Google Search or reddit where users are present a bunch of links. If they click on a link to see if it is interesting they destroy the current page. Hitting the back button 1) takes just long enough to make this expensive (sometimes really expensive) and 2) can result in the web page changing (such as on reddit) because the browser will reload the page. So users open the links in a new tab. What if when you clicked on a link in the Link List page it loaded in the Site View? How about loading an article on the right window and loading the digg comments page on the left hand side? The first week I had this in action and in use as my primary browser was mind opening.

Reading Lists

Watching how people use tabs you will notice an interesting usage pattern. Users are leaving a tab open because they want to come back to it later. This is often referred to as a Reading List and can also be called a complete failing our the existing bookmark system. For long term reading providing the means for users to quickly tag a page they can then close it. For example when researching a new car a user will leave open tabs that they find vary informative rather than making a bookmark folder. The second type of reading list is simpler. When doing a Google search a user might click on every search result in a new tab. The user will then read through each tab to see if it has the content they want. The browser should provide a way to quickly push links into a reading list rather then loading them in a new tab and conversely provide the means to pop the next item on the list or even clear the list. With the split views and reading lists users should be able to use the browser for a single task with only the two web views.

Buffers

An average human can only keep around seven things in their head at a time. Even if a browser can handle five hundred tabs the user is probably only thinking about the 493-500'th tabs. Each task view with the two windows is a buffer that can be layered. While split views and reading lists are great for a single task users often have several tasks such as reading news, gmail, and some Java docs. Providing buffers for each task will result on average around 7 tasks and 14 web views total rather then the tens to hundreds of unsorted tabs that users have today.

Ubiquity [http://labs.mozilla.com/ubiquity/]

If we take away the editable location bar how will the user go to a website? As mentioned before the location bar is a widget that has become smarter over time, but is still little more then a crude command line interface into the browser. The Ubiquity projects shows a much better route moving forward. You hit ctrl-space and start typing. A window that could take up the entire screen provides rich feedback about what you are typing. You can do all of the classical location bar commands such as manually typing in a url, 'go slashdot.org' or searching Google for foo 'g foo' or Wikipedia 'wiki foo'. But you can also have a quick calculator, translator and many others. Because it is text based it is extremely rich allowing you do do many things and can be easily expanded upon. [ See ubiquity for more info as they have spent time writing up nice simple descriptions and examples ]

Temporary-Buffer

Another reason users often open a new tab is to do trivial one off things such as typing a word into google to spell check it, going to whatismyip.com. While ideally these should be Ubiquity commands the user could cause a single web view to appear that floats on top of the buffer that the user can use for a short time before returning to the existing buffer.

Html Configuration & No Dialogs

A browser has a fair number of configuration options and yet due to the classic assumption about how configuration dialogs are presented there is no way to search a configuration dialog. Moving the browser configuration dialog into the browser and making it a web page gives all of the features a browser presents including easy searching and the ability to present many options in a scrolling interface.

JavaScript alerts, downloads, file dialogs all of these can be embedded into the web page. Popups will be local to the web page and while they might be able to block the page they will never be able to block the while application. A user giving a demo on buffer 2 will never have a popup blocking him telling him to accept a security cert for the site that is loading on buffer #1.

When access the MySite for the information about a web page would be flipped over and on the back would be the site configuration and information about all of the data, cache, history graph, and more. By flipping over the page you are not limited to a small dialog, but get the full size of the screen.

Guest Mode

Private browsing mode fails for the basic fact that we tell users to use it when they want to hide something. Users of course forget to use it from time to time resulting in them deleting their complete history out of fear. And to top it off when they are browsing porn because our browsers don't have their history and so we can't provide basic services such as letting them search their history for that video they really liked last week. Fail all around. What users really want is if someone uses their browser they will not see something. Given that our data is encrypted you must authenticate yourself on start any time you are not authenticated you are in guest mode. Really guest mode and private browsing mode are the exact same thing, but used differently. With private browsing you use it when you want to hide something, with guest mode you use it when someone is watching or when someone else is using your browser.

Meta Information

On top of the two windows is a meta bar which depending on the OS can be in the window title bar providing maximum web page space. This meta bar is created dynamically and while a default is provided users should be encouraged to customize it by making it trivial. It could be the classic back forward button, URL line edit, and search, but by default it could be the site title and the widgets that you often find in the location bar such as the RSS icon, the site icon, and network information such as if we are using SSL. The URL of the site could be a band at the top of the page that scrolls with the web page only needed at the start to confirm the site you are on.

Command Reporting

If you disable cookies, use adblock, or click to flash there would be a way for the user to see what the browser has blocked to easily unblock.

Plugins

A browser is only as rich as the extensibility that is provided by the extension framework. There are too many features that can significantly enhance the browser for a user that should not be part of the primary application.

The replacement for History or Bookmarks: MySite

MySite is the collection of all information about a site stored together where they can be more useful and provide richer features.

History and bookmarks are really one and the same and it is time we tried to keep the separate. History is sites you have been to, bookmarks are sites you have been to that you want to remember. As you move beyond two basic lists, the features quickly overlap. When was the last time I visited a bookmark? Can I sort a bookmark folder of links based upon when I visited them? What did the web page look like when I bookmarked it? Based upon my history what sites do you think I should (in the classical sense) bookmark? Every time you visit as website there is all sorts of information the browser can store to be useful for a user. To start with simply keeping the cache of the web page. This lets the browser:
- Full text search of every site the user has visited in every version.
- Offline browsing of sites they have been to
- Provide screenshots that can be used for all sorts of tools like a "Top Sites" page and a rich visual history browser.

Given that users have TB hard drives why are browsers limiting their cache to 50MB? Why not 50GB? Beyond just making day to day browsing faster having a full history can be very valuable. Research has to be done to see how much would be of value, but just defaulting it at 50MB seems foolish.

On the data side all data for a site should be stored in a single place. Rather then having a cookie jar that handles every site out there design a cookie jar that can be cheaply created for each new site that stores the cookies for that site. Deleting all the data for a site could be as simple as deleting the entire cookie jar (file/folder/database) with no possibility of bugs. With a cookie jar that only has the cookies for site X there is no possible way for a script from site X to grab your bank.com cookies in a hidden iframe or anything.

Each site should have the ability to customize every browser configuration for that site. Today some browsers let you turn off a few things such as cookies for a site, but why stop there? Every single configuration option related to the web page should have the ability to be customized on a per site bases. Cookies, custom stylesheet, zooming factor, fonts, etc. This is done in a way that is cheap so each tool and addon (see adblock, and many similar tools) don't need to re-invent the exception/whitelist dialog.

MySite should store the URL's that were browsed including the referral URL and meta information such as how they left the site (click on a link, went to a bookmark or closed the tab), where was the scrollbar. This valuable information is small and can provide rich history/replay browsing later.

Rather then bookmarking a site, a user will tag a page with a bookmark. Free form tags on any page allows a user to not only quickly mark a page, but also add meta information about why they tagged it. Pages can have multiple tags and searching for tags is the default method for accessing tags. Tags are cheap, the user doesn't have to decide to bookmark a page, there is no tag management or bookmark menu tree to keep clean.

How valuable is your browsing history? Browsing history is literally your thoughts in computer form. Walking through the history of the CEO might tell you that he has been spending a lot of time reading up on a some small startup. Is he looking to buy it? What if that CEO was Steve Jobs? How about going through the average users history? You will discover where they are thinking of going on vacation, what car they might buy, where they get there news and probably what fetish they have. What if you saw someone was searching for "How to make an atomic bomb", "How to kill my wife/husband", spending time looking at Monster.com, a medical sites, a forum site where they post under an alias. This is pretty private information. If you were to sell a diary application what would be the first thing you would offer? Encryption. Given the amount of time users spend in the browser and how sensitive the history data is it is insane that browsers don't offer full encryption by default.

[http://www.chromium.org/chromium-os/chromiumos-design-docs/protecting-cached-user-data]

MySite has to parts, cachable and non-cachable. You can (usually) re-download a webpage, but you can't re-download your cookies if they are deleted. The non-cachable parts should be in a form that is sharable so they can be on multiple computers. From multiple computers, to reloading an OS what is the value of having a history that can hold 100 years of links if every six months they reload Windows?

While initial I did say that you should remove all of the classical UI elements that a browser has all of the backend changes could support the classical view, but that would destroy a lot of the value of those changes.

Further notes and ideas:

More Meta Information
There are various history tree viewing examples out there such as: https://addons.mozilla.org/en-US/firefox/addon/13316/

Direct History Views

From the Summer09 Mozilla labs design challenge TabViz
http://design-challenge.mozillalabs.com/summer09/

Managing multiple Qt versions with Git clone

2009-07-05T16:06:00.013-04:00

Like many Qt developers I have many different versions of Qt on one computer. In the past I used to have a copy of nearly every tarball and a built version. My /home/ben/dev directory looked something like this:

...
qt-all-opensource-src-4.5.0/
qt-all-opensource-src-4.5.0.tar.bz2
qt-all-opensource-src-4.5.1/
qt-all-opensource-src-4.5.1.tar.bz2
qt-all-opensource-src-4.5.2/
qt-all-opensource-src-4.5.2.tar.bz2
...

Now that the Qt repository has been open for a while we can take advantage of a nice feature of git clone called local cloning. On my computer I have a primary Qt repository (/home/ben/dev/qt-master/) which has has remotes pointing to my personal Qt fork and the main Qt repository.

Rather then downloading a tarball of 4.5.2 I now do a local clone of Qt and then checkout the tag "v4.5.2". Although Gitorious does not list tags (missing feature?) both Qt 4.5.1 and 4.5.2 have been tagged in the Qt Git repository.

git clone -l qt-master qt-4.5.2
cd qt-4.5.2
git tag -l  # List the available tags I can checkout
git checkout v4.5.2

A local clone will share object files when possible saving you a lot of hard drive space. For Qt versions older then 4.5.1 I still have the tarballs, but going forward there are a lot of advantages to using this beyond saving space. In the past when tracking down a bug in a specific release I have often tweaked the code or applied a patch. Now that the releases are fully backed by Git I can quickly revert back to the release as well as using tools like git blame and git log to get much more information about any files I am investigating. You also get the tests and util directories which are stripped out of releases.

In addition to the releases I also have a clone for the Qt 4.5 branch, what will be 4.5.3. qt-master is the original Git repository which tracks what will become Qt 4.6. After this change I ended up with a directory like this:

...
qt-all-opensource-src-4.5.0/
qt-all-opensource-src-4.5.0.tar.bz2
qt-4.5.1/
qt-4.5.2/
qt-4.5/
qt-master/
...

Like many Qt developers I have written a bash script 'qs' that I have in my shell profile to quickly list and switch between Qt directories.

'qs' - Display current Qt version
'qs ' - List all qt directories
'qs qt-4.5.2' - Swap environment to use Qt 4.5.2
'qcd' - Change to the current Qt's directory

This Git clone trick of course applies to any Git based project, not just Qt, but it is yet another hidden gem of the decision to open the Qt repository.

Open Qt repository and hidden gems

2009-05-13T19:10:00.003-04:00

On Monday Qt Software launched the public repository for Qt. Already there are more then a dozen patches waiting for review in the merge request list and no doubt there will be many more. To help developers submit patches that are higher quality and more likely to be accepted a number of documents have been published on the Qt wiki. Beyond the wiki in the source repository there are new things that could not previously be found in the snapshots or releases. With so many new things I wanted to highlight a few of the items I found most interesting.

Qt Coding Style Followed not only by Qt, but by other projects such as Arora and applications in KDE. Having a common coding style makes code easier to read and review. In the past few years I have seen many applications adopt this style and I look forward to seeing the usage grow.

Api Design Principles An updated version of the old Qt Quarterly article Designing Qt-Style C++ APIs from the first paragraph: "This document tries to summarize the know-how we've accumulated on designing Qt-style APIs. Many of the guidelines are universal; others are more conventional, and we follow them primarily for consistency with existing APIs."

The Little Manual of API Design by Jasmin Blanchette’s In the wiki page for Api Design Principles you find a link to the this pdf which is now public. This first rate book should be manditory reading by any programmer. While many of the examples are from Qt and in C++ most of its ideas apply beyond Qt and C++. Rather then doing it a disservice and trying to describe it, here is the table of contents which gives a good overview.

1 Introduction 5 
2 Characteristics of Good APIs 7 
2.1 Easy to learn and memorize . . . . . . . . . . . . . . . . 8 
2.2 Leads to readable code . . . . . . . . . . . . . . . . . . 9 
2.3 Hard to misuse . . . . . . . . . . . . . . . . . . . . . . 10 
2.4 Easy to extend . . . . . . . . . . . . . . . . . . . . . . 12 
2.5 Complete . . . . . . . . . . . . . . . . . . . . . . . . . 13 
3 The Design Process 15 
3.1 Know the requirements . . . . . . . . . . . . . . . . . . 15 
3.2 Write use cases before you write any other code . . . . . 15 
3.3 Look for similar APIs in the same library . . . . . . . . 16 
3.4 Deﬁne the API before you implement it . . . . . . . . . . 17 
3.5 Have your peers review your API . . . . . . . . . . . . . 17 
3.6 Write several examples against the API. . . . . . . . . . 18 
3.7 Prepare for extensions. . . . . . . . . . . . . . . . . . 18 
3.8 Don’t publish internal APIs without review. . . . . . . . 18 
3.9 When in doubt, leave it out . . . . . . . . . . . . . . . 19 
4 Design Guidelines 21 
4.1 Choose self-explanatory names and signatures . . . . . . . 21 
4.2 Choose unambiguous names for related things . . . . . . . 22 
4.3 Beware of false consistency . . . . . . . . . . . . . . . 23 
4.4 Avoid abbreviations . . . . . . . . . . . . . . . . . . . 25 
4.5 Prefer speciﬁc names to general names . . . . . . . . . . 25 
4.6 Don’t be a slave of an underlying API’s naming practices . 26 
4.7 Choose good defaults . . . . . . . . . . . . . . . . . . . 27 
4.8 Avoid making your APIs overly clever . . . . . . . . . . . 28 
4.9 Pay attention to edge cases . . . . . . . . . . . . . . . 28 
4.10 Be careful when deﬁning virtual APIs . . . . . . . . . . 30 
4.11 Strive for property-based APIs . . . . . . . . . . . . . 31 
4.12 The best API is no API . . . . . . . . . . . . . . . . . 32

Codeing Conventions and Binary Compatibility Workarounds Two good documents that walk through a number of issues that Qt has to deal with being a library that is used on many platforms and compilers. An interesting read for anyone who uses C++ or Qt.

Autotests and Benchmarks After several years of discussing it, the Qt autotests have finally been published. Everyone in KDE can finally checkout how class Foo is tested and when writing patches for Qt an attempt should always be made to write a matching autotest. See some interesting usage of qtestlib and get plenty of example of how to test gui widgets. For those who are writing their own tests check out my QAutoTestGenerator tool which generates test stub files for you to help get you started. Hopefully having this large set of tests open will encourage more Qt projects to have tests. For those who are using Git and qtestlib I have written a git hook that can be found in the Arora repository that will run the matching autotest for files you change to check for any regressions before commiting.

The full commit log for Qt While in the past the qt-snapshot git repository gave a one day granularity the new repository includes every commit with the commit log. The other day while tracking down the QTemporary bug in 4.5.1 I wrote a little test application and Thiago used git bisect to find in twenty minutes the exact commit that had caused the breakage.

git bisect start HEAD v4.5.0 -- src/corelib/io
git bisect run sh -c 'LD_LIBRARY_PATH= make sub-corelib && (ulimit -n 250; /tmp/qfile-bug/qfile-bug)'

With the full commit log tools like bisect and blame become even more useful for inspecting the Qt source code.

With the Qt sourcecode on gitorious It will be easier for developers that need a bug fixed to help contribute and get it fixed in Qt faster. Those who use Qt and want to understand how something works or why it was developed in a particular way will be able to more easily explore the code base. And maybe KDE can even remove the legacy qt-copy. No doubt many good things will come out of the open Qt repository and I look forward to seeing what the community contributes.

PlanetQt.org launch

2009-03-18T22:50:00.000-04:00

After more then a year of thinking about doing it I finally setup planetqt.org and after getting the initial kinks out am officially launching it. While KDE is a major user of Qt there are many applications and developers out there who are not part of the KDE project or who's blogs/news really don't really fit on planetkde.org (or the freedesktop planet). Because of this there are a lot of really interesting Qt projects/tools/stuff out there that people don't know about. So that is why I started Planet Qt. Being made up of people I thought the frame from the Qt4 dance seemed to be a nice logo.

Submit atom/rss feeds for blogs about Qt development or the news feeds of applications/libraries that use Qt and I will add them.

Enjoy!

Tiling An Image With FractalFill

2009-03-02T21:50:00.002-05:00

Last summer Adam and I looked into how to speed up tiling an image. In particular we were using QImage (Not a QPixmap so X11 isn't involved in this) and tiling an image on it. This is a pretty common operation, especially on the web where many web pages have repeating backgrounds on items. We came up with a few different ways to do this in Qt.*

Naive

loop over the height 
  loop over the width
    fillRect(tile)

QPainter::eraseRect

Set a background brush to be the tiling image.
call painter.eraseRect(QRect(0, 0, image.width(), image.height());

QPainter::fillRect

call painter.fillRect(QRect(0, 0, image.width(), image.height(), brush(tile));

FractalFill

painter.drawImage(0, 0, tile)
while we haven't reached the width and height
  copy everything we have done so far to the next location

Benchmarking this we discovered that eraseRect was pretty horrible, but much more surprising was that our FractalFill algorithm that we tossed together was faster then all of Qt's method. (If any graphics ninja's want to suggest a even better algorithm we are all ears) Adam blogged about this some of the Trolls responded saying that Qt's painting had improved in a private branch which would be part of Qt 4.5. Now that Qt 4.5 is almost out I thought I would revisit to see how the painting performance numbers shaped up to be.

Along with the four algorithms there are two more that are added to the benchmark (Colored yellow in the graphs).

A base line, something that we shouldn't be able to be faster then: just calling painter.fillRect(QRect(0, 0, width, height), color);

Using the FractalFill to paint the whole image with a color.

First up is RGB16 with: 1000x5000

Qt 4.5 really has improved upon Qt 4.4. The problems in eraseRect are gone and everything is faster across the board moving to 4.5 and even better when using the raster engine. But more surprising is that FractalFill is still faster then the two QPainter functions.

Same format (RGB16), but in a larger size: 5000x5000

Again it is clear that eraseRect has been fixed and overall painting has been improved slightly.

Moving on to ARGB32: 1000x5000

And in 5000x5000

On this last one with the larger image sizes I started getting larger differences in the tests results. On ARGB32 the performance seemed to be about the same or slightly better.

Overall performance is much better in Qt 4.5 then 4.4. A big congrats to the Trolls for all their hard work. Painting speed has improved across the board decreasing the runtime of all of the functions. FractalFill is still faster then using QPainter's fillRect or eraseRect so for the time being if you looking for something faster then Qt's fill functions for tiling this might be useful for you.

Notes:
I have uploaded the source for the test application to a GitHub repository. I ran the application multiple times on my computer to generate the numbers using the release version of Qt, building from qt-snapshot with 4.3.2 on ubuntu and qt3compat turned off. Now that Qt 4.5 is out it would probably be worth converting to the new benchmark tool. I used Google's graph service to generate the graphs.

* Analyzing each algorithm to determining their Big-O notation is left as an exercise to the reader. ;)

Arora 0.5 and Oxygen Style

2009-02-20T21:32:00.004-05:00

Tonight I tagged Arora 0.5. And with every release starts development on the next release. One of the things I would like to improve for 0.6 is integration with KDE4. To help do that I have started a wiki page of KDE4 items. When using Arora on KDE a bunch of the first issues everyone runs into are really just improvements that need to be made to the Oxygen style. Little things like returning the KDE icons for back/forward, returning the correct size check boxes in QtWebkit, and supporting the new QTabBar style hints in Qt 4.5 will really help Arora fit in better with KDE 4. Of course these fixes will benefit everyone who uses QtWebKit in their application and anyone KDE application that uses the new QTabBar features. So I am putting out a request for help. If you happen to have your box all setup for KDE development and have a few minutes to spare helping making Oxygen an even better style would be most appreciated.

Comments rating systems, a close look at giving points

2009-02-06T20:12:00.002-05:00

On many social news sites users can give and take points on comments. The purpose of the points is to give hints about what comments are worth reading. But upon closer inspection this method often fails to provide a way to let the sites software sort or highlight good comments.

A lot of time and effort is spent perfecting the algorithm used to generate the order of stories, but when it comes to user comments on each article the trend the past few years has been to allow any user to vote up or vote down a comment and to sort on the values. This up down method seems to provide very little real value and produces very poor results as a means of sorting.

When a user sees a comment the basic version of the algorithm that users go through is something like this:

"If I see a post that looks good I vote it up. If I think it is wrong I vote down."

The number of points a good post gets is directly related to the number of users that read the post. The more users read the post the more are likely to award it points. A good post A posted at time 0 could get 100 points and the same post made at time +1hour will get 50 points and the next day will only get 3 points. At the same time a bad post B posted at time 0 can still get 8 points.

Or to sum it up with one question:
Is a comment with 47 points better then one that has 46 points? Frequently you will find the answer to be yes, no, and maybe depending upon when the post was made and having very little to do with the actual content. Because comments are not created at the same time the points that they each get can't accurately be compared without taking into account extra information. You could take into account the time sense the article was posted, the number of people who saw the comment, the number of people who voted on comments and the total votes given at the time when the vote was cast. All this does it make it more complicated so the same system can be used rater then asking of the system should have been used in the first place.

For many of the social new sites the comments were second fiddle to the main page. Once the jewel of the site the article sorting was working then comments were added. Many social news sites the past few years have employed some sort of up/down voting system for articles. Developers being developers decided to reuse this same code on comments. Lets users up/down vote on any comment and sort based upon value and blam job done in no time flat. Of course the article sorting isn't actually based upon only the value of votes, but takes into account time as a factor of the ranking.

It seems as though sorting comments was created out of the laziness of developers. They do have one thing going for them though. When a site is small there will not be too many comments on articles so users can quickly read all of the comments. For a small number of comments the sorting doesn't matter that much. It is only when you get a large enough base to get many good comments that the issue comes to light. Of course by that time if you have lots of users there is no real incentive to make the site better because the incentive of making something better to get users is diminished.

While social news sites think that their secret sauce is the front page I wouldn't be surprised if the comments page was of much more value. After playing around with the issue for way to long I ended up with a model very similar to the one slashdot already has. For any given comment it can be ranked from -5 to +5 and given a type. You end up with comments that have ranks like: +5 funny. +4 informative, -3 troll, etc. It takes very few users to agree that something is informative or funny and should get a value of -3 or +5.

The number of points a good post gets is directly related to the value the users find the post to be. This will results in the comment changing values less and less as time goes by (reaching its true value). A good post A posted at time 0 will have 0 points, but two minutes later will be at 3 and from then on some might say it is 2 some might say it is 4, but it quickly approaches what everyone feels the comment actual worth is. Compared this to posting the comment A after one hour and it will probably again be marked at 3 by a few users. This allows for much better comparisons between comments.

Combined with the small point scale is adjectives. This makes the user not just say "I like this", but have to think about what they like which results in much better scores. It also gets the added benefit of allowing users to do things like automatically give -1 points for funny comments if they find them annoying (or +2 in the opposite case).

To add to the debate there are several live examples of the different systems (often times they even cover the same article!). First going back to the objective which is to rank or highlight good comments for readers so they don't have to read everything. This includes getting rid of junk/spam/trolls. Now head over to slashdot and click on an article from yesterday. View comments by default it shows +4 and higher. They are pretty good and informative. Now go to reddit. While it is able to get rid of spam comments it displays everything having no such thing as +4 or higher and even if you sort by Top you still find that that non-informative, but early comment (which may or may not be really good) is right at the top with a huge number of points and you have to scroll around reading everything, attempting to find a post of any real worth. The good comments are just as likely to be near the bottom with 4 points as near the top with 400. When a social news site is unable to bring good comments to the viewer a good chunk of the value of the social new site is lost.

Having points for comments is a system that is so easy to implement that it will probably never go away, but a closer inspection shows that it really doesn't do a good job of the basic feature it was created for which is to allow the system to give hints about what comments are worth reading.

While I doubt many sites will change what they are doing this will hopefully spawn some conversations for developers who are creating the next generation social new site. While the slashdot system might not be the best we need to take a step back and look to see what the objective of ranking comments and how to best achieve that.

Photo by dougww

For another fun comparison blog like this one see the blog post that discusses two different ways to find the most attractive girls.

And for a blog discussing problems with giving users infinite karma points checkout Joshua Gross's Qualifying “Web Karma”: It shouldn’t be a game.

Update: Reddit has switched to a new comment system that attempts to resolve several of the problems listed. Read more on their blog: reddit new comment sorting system

Chromium On Linux (Part 2)

2009-02-01T05:00:00.000-05:00

I have been following the Chromium development the past six months as Google attempts to get Chrome working on the non-Windows platforms. In the process I have watched as they rediscovering that creating cross platform application from scratch is hard. Following up on my first blog entry on why you don't have Chrome on Linux today as a Linux user recent Unix platform discussions are just too amazing not to share.

On OS X the plans is to redo all of the UI and use native Cocoa API. There seems to be very little debate on this as there are many positives including a strong consistent API with a growing fickle user base. They still plan on doing their own window painting and not using the OS X theme though.

But on Linux is it a different story. Linux has very few users and those users like to use many different things and have different expectations. Rather then using a library one options would be to re-use the code from Windows. Chrome has an internal library called "Views" which wraps some Windows API/widgets and implements from scratch others. On the mailinglist Evan (one of the kick ass Linux Chrome devs) mentions how they are already duplicating work that already exists in Gtk.

"Pawel recently posted a patch to abstract out the views keyboard event handling for our "button that looks like an HTML link object". Well, there's a built-in one we could've used: http://library.gnome.org/devel/gtk/unstable/GtkLinkButton.html"

Not only Gtk, but Qt also has a widget that can do this. But this is just the tip of the iceberg as they are forced to port each feature from focus handling to widgets. The biggest thing though is that Chrome's tabs go all the way to the window boarder on Windows. On Linux this means to get this they would have to turn off the window manager decoration and do all the painting and window handling on their own. Even though they admit that "it's endless pain" they are pushed to do this for branding and experience reasons. One of the developers even suggested that this would be something for the Linux developers to do in their 20% time.

"What about rolling out [using the window manager] initially, then implement [drawing everything] as a 'group 20% project', the group being the linux team" (1)

Google developers have reported that many never actually use the 20% time contrary to Google's HR marketing and I guess it has gone to such extremes internally that now the 20% time is thought of as time you get to spend doing project tasks.

Working on Arora I planed from the start to have Arora working on different platforms. Even switching between Gnome and KDE icons, button order, shortcuts and more will change. Looking through the screenshots you can see that Arora does a decent job of integrating with each desktop. Many Linux users are geeks, the very type that would download Chrome's source, improve it, write plugins and more. Actively not wanting to fit into their desktop doesn't seem the best route to take.

While they can always change their mind, for now the push in Chrome seems to be to do their own window boarder painting and handling, porting the views (creating a new cross platform toolkit?) and not bothering about real integration with any Linux platform until later.

Or to sum up in the words of Ben Goodger:

"Yes there will be some number of people that squeal. But honestly how bad will it be?"

1) http://groups.google.com/group/chromium-dev/msg/9146805bb27ba30c?dmode=source
2) http://groups.google.com/group/chromium-dev/browse_thread/thread/b89ab99a0c848b89#

Qt 4.5 will be released under the LGPL.

2009-01-14T03:20:00.001-05:00

This morning Nokia/Trolltech announced that starting with version 4.5 all versions of Qt will be also released under the LGPL 1.2.

This bit of news is absolutely jaw dropping amazing. Game changing amazing.

Near every scenario where toolkit X was chosen over Qt now gets to be rethought. Can't afford a license? Boss wont let you buy it or even get a trial because of the cost? Only allowed to use software that you don't have to pay for? Only want to use LGPL software for philosophical reasons? In all these cases now you can use Qt. No doubt there will be considerable amounts written about this change in the coming months.

Shortly after I first got into Qt, Gnome was formed in response to KDE and in particular Qt's non-free licensing at the time. A few months later Qt was released under the GPL, but Gnome and Gtk had picked up steam and continued to be the projects they are today. Part of that was that GTK+ was released under the LGPL. So for the last twelve years I have unfortunately heard many discussions more about licensing then about the technical nature of the libraries. After all this time for that wall to be abolished is stunning. So many years of licensing and wham, in one day it is gone! The emotions will still be there and it will take time, but now that both libraries are under the same license hopefully developers will be more willing to combine efforts and choose tools no matter what toolkit they use.

Developers and startups that want to make applications that they hope to eventually make money while keeping the source closed will be much more open to the idea of using Qt. In the corporate world having Qt under the LGPL will allow developers to bring their Qt experience into the office to get jobs done and show off the capability and power of Qt without having to go through the hassle of getting a trial license. Companies that don't want anything to do with GPL or LGPL can of course still buy a commercial license.

From QString, QtWebKit, QtConcurrent, and Designer Qt has many useful components and it will be interesting to see how people use it in the future. This license change will no doubt open many more doors for Qt. Combining free, high quality, and cross platform is a killer combination. Qt's API is fantastic and the documentation is first rate. This change will only cause Qt's usage to increase across the board and the need for developers with Qt experience to increase. A solid way to build applications on every platform and what matters most for many, free.

I have no doubt that interest in QtWebKit and my own little project Arora will increase. Putting Linux with Qt Embedded and WebKit for free is a slam dunk. Interest in WebKit has been growing by leaps and bounds the past year and it will only increase more in the next. QtWebKit backed by a LGPL Qt provides a very attractive solution for many. Only time will tell what happens of course.

From playing around with Qt back in 1996, being part of to KDE, creating applications for the Sharp Zaurus, working for Trolltech on Qt itself, and contributing to WebKit it has been an amazing twelve years and I look forward to see where it goes in the next twelve.

The status of Chromium on Linux

2008-11-02T16:48:00.005-05:00

When Chromium was first announced in the beginning of September I was very surprised that it was a Windows only application given that WebKit is very much cross platform. The past few weeks I have been spending a little bit of time here and there hacking on the source code and thought I would write an update for those who are interested on the status of the native port of Chromium on Linux.

On day one you could checkout the source code on Linux and you could build some things. Of course all that you were building were some object files, nothing more, not even Webkit was being built. There was no test application, no linking and no Chrome.

From what I can tell nearly all of the development for Chrome was done on Windows in Visual Studio. There is even a c# tool that can be found in the sources. This lead to the case where the normal course of action when something didn't build on Linux was to just disable it. So by the time that the release was made nearly nothing was being built. I am also pretty sure that the Chromium port was entirely different then the Android port

Starting with the glue directory I went file by file fixing the compiler errors. Developed in Windows there was fun fixes such as the Windows "String.h" include that was not used, but caused build breakage on non-Windows platforms. Many patches later a lot more builds and Linux. Linux is part of the build farm so as each file was fixed and enabled it became one more file that Windows developers could not break or their change would be reverted. The Chromium developers had also been syncing up with WebKit trunk and so the Chromium fork of WebKit also built. Attempting to link a test application we found only 500 missing symbols. Tracking these down one by one and either porting, fixing or adding a stub the test application finally linked about a week ago. It can't do anything and will crash, but that is where we are. (update: Pointed out by Evan as of Friday it can render google.com.)

Beyond a webkit test application, Chrome on Linux is far away from happening. There are still a large number of basic components such as fonts, clipboard, plugins, and basic widget handling that have to be implemented. The test application is only a test for WebKit. In the next month or so you should be able to see WebKit rendering and at some point next year a beta of Chrome on Linux will be running, but it wont be tomorrow or next week.

Sharing some of the same code OS X is in the same boat, but they are a little better because they can re-use the OS X WebKit code, they currently can render pages, but can't do things like click on links.

Things are not as bad as they sound. The Chromium team is a good team that is moving forward quickly. As they integrate more with upstream WebKit more of the Linux port will work. They are also re-writing chunks of their code to be more cross platform friendly. And now that OS X is kinda working and Linux links there is a lot more interest in putting developer time on the Linux port to get things done.

An important lesson that Google should take away from this is the importance of developing cross platform software on all platforms at the same time. Assigning just one guy six months ago to the Linux port would have significantly improved the source on day 1. Having the build bot catch errors early such as using a Windows only function rather then a C function pays off in developing cross platform code. The sad thing is that any company that does cross platform development could have told them this without them having to learn it the hard way.

If you are interested in helping with Chromium and want to hack in git rather than on svn, an official mirror that is updated every other minute is now on github:
http://github.com/chromium/chromium/tree/master

I have written up getting started instructions and created a script to upload (and update) patches from git here:
http://github.com/icefox/chromium_tools/tree/master

Thanks to the guys on #chromium-dev who are very patient and helpful.

Git Hooks

2008-10-17T02:40:00.001-04:00

Git hooks are scripts that are run by Git before or after certain commands. Because the hooks are run locally and not on the server it allows for a lot of freedom to write more in depth scripts. The scripts are located in the .git/hooks directory and Git comes with some sample scripts that can be enable by removing the .sample suffix and making it executable.

A nice example is the included pre-commit script which checks for trailing whitespace in the patch and exits with 1 if any are found. If a script exits with non-zero then the command is aborted.

Below is a simple, but very useful script. All it does is run make in your current directory if a Makefile exists. If make fails then it returns non zero preventing the commit from occurring. While this script is crude and verbose, it works in preventing commits the break the build. Compared to a server side script which has to build the entire project from scratch on potentially a slow machine, there should be nothing to build (you did run make before committing right?) so it should return very quickly. With this one script you can prevent most build errors from getting pushed up to the central repository and causing problems for other developers. One liner "fixes" that break the build can be a thing of the past.

pre-commit_make


#!/bin/sh
echo "--Attempting to build--"
if [ -f Makefile ] ; then
    make --quiet 
    if [ $? != 0 ] ; then
        echo "--Build failure--";
        exit 1
    fi
fi
echo "--Attempting to build pass--"

And really you can simplify it down to just this

 
#!/bin/sh
[ ! -f Makefile ] || make

Files in the .git/hooks directory are not part of the repository and so they are not tracked. A workaround is to have a git_hooks directory at the top of your repository like done in Arora and symlink .git/hooks to git_hooks whenever you clone. This way the hooks will be part of the project, under revision control and accessible to everyone.

When using git commit if you want to skip the hooks you can pass the --no-verify option.

Although many hooks you see are written in shell script they can be in any language don't even need to be a script, just an executable in the hooks directory.

An important thing to remember when writing hooks is that the patch and the list of patched files is available. A script that validates XML shouldn't do anything if the change doesn't modify xml files. 'git diff-index --name-only HEAD' is one simple way to get the list of all the changed files Using this you can make sure that no the hooks will always run quickly and only test/check what is needed.

There are a number of different hooks that git provides access to. For a complete list and details about what each of them are for checkout the Git hooks documentation.

Here are some hooks that I have either thought about doing, have already done, or found via Google:

pre-commit
- Check that the project compiles
- Check for compiler warnings
- Check that the auto tests for the files that are being modified still pass.
- Check that the performance tests for the files that are being modified still pass.
- Check that the code in the patch conforms to a code style
- Validate XML files
- Spell check user strings, documentation, etc
- Check for binary incompatibility
- Try to build a release package
- Check that any public API is documented and has no errors
- Verify any new files have a copyright header
- Check that all public strings are properly wrapped so they can be translated
- Check for calls to functions that are known to be deprecated or inefficient.
- Check for swear words
- Reject commits made between 4am and 7am with a note to go to bed.
- Automatically remove whitespace
- Run RSpec tests before allowing a commit
- Some rake testing

commit-msg
- Spell check the commit message
- Verify an agreed upon commit message format is used.
- Prevent business announcements or key words from leaking out in a message.

pre-rebase
- Useful for preventing an accidental rebase from occurring on a master branch that everyone shares.

post-commit
- Send out an e-mail to the mailing list
- Build a package
- Build/Update API documentation for the website
- Update/close bugs in a bug tracking system
- Spawn off builds on other operating systems

post-receive
- Update tickets in Lighthouse

I have mostly played around with the user side where each commit is one patch and life is pretty simple, but on the server a push can contain a number of commits and the scripts are only called once. Checking out the docs and the scripts in the git source code contrib/hooks directory for more information on how to extract each commit.

Taking advantage of the hooks system one can easily add checks for all sorts of things that can be automated resulting in always building source that is higher quality, more consistent, and with less embarrassing errors. The ease that anyone can add hooks will hopefully cause more and more projects to utilize this built in part of git.

If you have written an interesting hook feel free to post a link to it in the comments for others to checkout.

Hook photo taken by *L*u*z*a*

LSDVD

2008-10-07T17:34:00.001-04:00

LSDVD was the name of a successful project to create a fully functional software DVD player for Linux back in 1999.

At the start of my freshman year at RIT in 1999 A guy by the name of Gad gave a presentation about the DVD format and his project to create a software DVD player for Linux. This was all back before DeCSS, nearly every DVD related tool that exists today and really before Linux was used by many as a desktop machine. I didn't even own a DVD reader for my computer, but it sounded like an exciting project so I picked up a 2X dvd drive for $109 and ordered my first DVD online.

I got involved in the project by working on the graphical front-end for the application. Over time I worked on many areas of the project, but I was in charge of the front-end from then on. We worked on and off on the project while going to school and through the winter, but the project was going very slowly. RIT runs on the quarter system and so in the spring took time off and rented a house north of RIT in Greece so we could work on the project full time. Day in and day out we worked on the project. Moving into the house with only a few hundred dollars to my name I slept on a couch in the basement and ate what canned food and Ramen noodles I had. We bought wood from Home Depot and built ourselves some desks for our computers and placed them in the living rooms. The house was pretty much empty other then a kitchen table and a mattress in one of the rooms. We would wake up, sit at the computer and code till it was time to sleep. Highlights included when someone would come over and we would have a real cooked meal (and watching office space).

Before DeCSS there was very little selection for DVD's to test because we could only read DVD's that were not encrypted and region free. This turned out to be mostly old gore flicks. It was a nice welcome when we could finally access our DVD collection to test against. The other two guys in the group were Paul and Dave. Dave was a crazy hacker who was in charge of the video decoding. Paul was in charge of the project. He set us up with the cvs server, managed the build script, all the small things that a project needs to survive, and of course huge chunks of the DVD player itself.

When we first got the MPEG2 decoder written it was still slow and so it would que twenty seconds of frames and then beep several times before playing it. We had a computer in the corner of the room playing Enemy of the State. Every five minutes it would beep and we would all run over to watch the video play. It took at least a day or so to get through the movie.

At each stage there was a fun magical moment as it came together for the first time. Getting audio working for the first time. Getting the audio and video sync working, subtitles, menus, and more. And of course we had to do testing too :)

Gad was looking for a company to support us and luckily one decided to buy us. Two months after we had moved into the house in Greece, we were acquired by a company in California. The four of us had never officially formed a company in that short time and so in trade for the code we got a small sign on bonus. Part of the agreement was that we had to live in California (which they paid for) and we were all excited about that. Flying out to California we spent the summer working on the DVD player. I have many good memories from that trip. We got to see a bit of California and went to a number of trade shows. I really grew as a programmer on this project, learning quite a bit from the other three especially Paul who designed the system. It was my first real large-scale project. Returning to Rochester that fall we continued to work on the player and got it to almost release status, but alas the project was killed in April 2001 due to financial problems within the company. It was less then a year later when the company was sold. The only one who have the application now is a small server sitting somewhere in the new company with our CVS code repository on it.

LSDVD includes the following features
* Almost perfect (within 1 frames) audio/video synchronization.
* Ran smoothly on a Celeron300a.
* Full menu support including subpictures.
* Graphical frontend.
* Didn't need to mount the media before playing.
* Worked on SMP machines.
* Fully supported AC3/PCM audio.
* All of the features that you would expect a stand alone DVD player to have such as seamless branches.

Whenever I see how little the software dvd players on Linux can still do and how much CPU they require I always have to smile.

Below is the one screenshot I still have of the application which was our easter egg and team photo.

My hacker bag

2008-10-07T00:43:00.005-04:00

In March of 1999 a Belgian bag manufacturing company created a contest to promote their new line of "Hacker" clothes. The first 99 people to guess the login and password to their contest site got a "Hacker" bag. This was an interesting enough of a story to get it posted on http://www.slashdot.org which is where I saw it. I began to work on the problem and with the help of several others eventually got a bag.

The way that you found out if your login and password was correct was by a JavaScript program embedded in the web page. Having done some JavaScript the previous year I figured that it would be pretty easy to modify the page and JavaScript code to go through every possible combination until the winner was found.

Pretty soon I knew that the JavaScript engines of the day would be way to slow and a C/C++ version was needed. Later that afternoon when I got a version working I posted it on the discussion forums on Slashdot. Several other people were interested and began to help improve the application. A whole week with little sleep went by while we spend time optimizing the code, making sure the code was creating correct output, calculating how many years it would take to complete, and chatting on IRC.

By the middle of the week not much had changed. On the website they hinted that the username was 16 character long and the password was 4 characters long. The backpack that you would win was called "Host" and so we were almost sure that was the password. We planned a couple of wild guesses that might give us the answer for the login. The first idea was that the subscript on the packs was edgyhipunique or something like that. Although it was 16 characters, it was not the solution. So we went further. Next we tried all combinations of the first 16 letters of this sentence. All combinations were checked out, but nothing was found there either.

As we were continuing to make speed improvements a web site was set up to track what blocks people where checking so that people wouldn't check the same block twice. In all over 500 people were helping out in checking for the answer.

In the end someone went to one of Kipling's stores and figured out that one of the bags ("mailbomb") serial number appended with 001 was in fact the login.
Login: 9840112000309001
Password: host
In the end the brute force solution didn't find the login and password, but it was fun and I submitted the answer anyway. To my surprise a few months later got a bag in the mail. While the bag was neat it was the thrill of hacking on the code that I enjoyed much more.

I found the source tarball on a backup CD and have put it up online if anyone cares to look. It includes some binaries which probably don't work anymore and some messing around with the javscript too. Looking at the code I am horrified at it, but I was very happy that when I ran "./a.out -login 9840112000309001" it was quickly able to find the winning combination.

Javascript speed, the browser wars, and the death of IE6

2008-09-27T23:34:00.006-04:00

Over the past year the FireFox, WebKit, and now Chrome teams have been going back and forth with faster and faster Javascript engines. Shortly after each update a number of blogs run all the engines through the benchmarks. When Internet Explorer is included it is always in last place. Not just by a little bit but by a significant amount. It is so slow that it is often just left out of the graph altogether. The IE team is working on it and the latest beta of IE 8 is three times faster then IE7, but it is still three times slower¹ than Firefox 3.0.1. And sense that article FireFox and everyone else have gotten even faster engines, not just by a little, but by a significant amount. This isn't about IE8 or IE7, but about IE6 and its market share. IE6 currently has around 35% market share which is a huge number of users to fight for. All of the Javascript wars is about making those AJAX and heavy Javascript applications run faster. With developers using fast computers with Firefox one can see what is going to happen when someone on a 800Mhz computer and IE6 tries to load the site. Already I have heard of developers migrating users off IE to FireFox because it would take the user several minutes to just load their Javascript heavy site. In the next year as developers take advantage of the new speed this huge swath of users will find themselves feeling more and more pain. Developers might try speeding up a thing or two, but the speed difference between the engines will only allow a developer to do so much and many developers will simply ignore the problem. Sure users can still *use* the site if they are willing to wait, but eventually they will hear about how their friend doesn't have the same problem with FireFox/Safari/Chrome/Arora and upgrade. While tabs and add ons have caused a downswing in IE's marketshare it might be javascript performance that will be the tipping point for the end of IE6. When FireFox got tabs you could still browse the web with IE6, but when FireFox gets a much faster Javascript engine you might not be able to browse the web with IE6.
1

Adobe owns the web and they don't even know it.

2008-09-15T17:43:00.001-04:00

The last ten months I have been hacking on a little cross platform WebKit based browser call Arora. Users try it out and are mostly happy, but near all of them end up asking me the same thing:

How do I get flash to work?

Arora uses the Qt and the current version (4.4) does not ship with support for netscape plugins. After the release of Qt 4.4 this feature was added to QtWebKit and will be part of Qt 4.5 this winter. To get it today all you have to do is compile Arora against WebKit trunk. Every few days someone pops into the Arora irc channel asking how they get get plugins. They say "plugins", but they really mean flash, and by flash they really mean video. Every day users are willing to download a development version of WebKit, built it and then build Arora just so they can get their YouTube fix. Users! These are the people who are not willing to do anything are willing to go through all that pain on their own. Big red alarms should be going off at the w3c that a binary plugin is required for browsing these days. Arora has pretty clear documentation on how to get plugins working so I can only imagine how many users went through the hassle and didn't have to ask for a pointer to the instructions. There was even a developer who after putting Arora on an arm device asked how to get Flash working. After discovering that he could not just copy the flash libraries to his device I got the distinct impression that no matter how good Arora was if he couldn't get flash it wouldn't be good enough. The majority of flash on the web for me seems to be 1) videos 2) ads 3) movie websites. I even had a user tell me that Arora was fantastic and would use it every day, but only once Qt snapshot had flash. In the last three years flash has gone from just another plugin to the only plugin on the web browser that matters. I myself never installed flash in Linux until two years ago. This is of course all because of video and specifically YouTube.

But here is where I am puzzled. Adobe is acting like they deserve to be the standard rather then what they should be which is scrambling to make sure they keep their foothold in the browser. Today flash is installed on nearly every browser, but "almost all" should not be good enough. They should have binaries for every platform under the sun, a QA team like no other, and paid developers on every open source browser to make sure the integration is perfect. Adobe should make sure that there is no geek out there who is fed up with flash and willing to dedicate their time to hacking on video on HTML 5.

Flash as a product is pretty bad. It frequently causes the browser to crash and Konqueror (and now Chrome) even ran flash in its own process so that when it crashed it wouldn't take down your whole browser. Flash ads frequently cause your CPU to spike and cause your page loading to freeze. As a user experience they are pretty bad to boot and until recently Google couldn't even index them. Adobe regularly make releases that break browsers and only release binaries for a insanely small set of platforms. Want to run 64bit Linux, or any BSD? No flash for you.

So what might the future hold? It was pretty clear from the Chrome presentation and literature that Google dislikes flash's presence in their browser. It cuts through their security model and in general doesn't play well with what the goals of their project are. But just like me probably found that their initial user base didn't care about their design goals, but just want to see videos of ninja cats. Video through HTML5 is real and coming. It will be in FireFox and WebKit browsers very soon. Up until now I hadn't thought much about this because of the YouTube problem. Even if I.E. supported HTML5 video why would YouTube go through the hassle of upgrading? With Google's Chrome there is finally a very compelling reason for Google to apply internal pressure to get support for HTML5 Video in YouTube. Once YouTube can stream those videos then Chrome could ship with plugins off by default (or disabled until your click on it in some fancy javascript way) creating a more stable, faster, and secure browsing experience. And what about those flash ads? Google uses text ads so they would be hurting competitors to boot. Once YouTube switches the other video sites will follow suit and copy this feature crushing flash's primary reason for existing. Of course only time will tell what really happens with YouTube.

Adobe has bet big on Flash and I mean huge. Six years ago they were a desktop company producing desktop software. But the past few years they have changed. They bought Macromedia and along with flash got their board of directors. For them flash, flex and the web is the future. They want to be the Microsoft of the web. They are directly completing with every cross platform toolkit out there in an attempt to create the new way that desktop and web software is created. They are riding the wave of flash installed and enabled on your browser and if that starts to change they will not be happy. If the golden goose (video) stops laying eggs before some killer flex apps are released there will be problems. Who knows how unhappy they were when the iPhone was released without flash and a dedicated YouTube application sat on the main screen. Perhaps they wanted too much money? There is a video on YouTube a short while back where on of the Adobe sales guys was giving a presentation at Google about flex. There was around ten people in the audience and near the end of the presentation he started saying

Just between you and me Adobe plans to ... platform .. web .. embedded .. flex

My jaw hit the floor when he did that and he was no doubt reprimanded if not fired when he returned to Adobe for the plans he spilled*.

No doubt Adobe will try to flight, maybe create their own YouTube clone, maybe fight with patents to keep HTML5 video off the web. Already they have announced that they will be removing the license fees on the next major releases of Adobe Flash Player for devices. You can read it for yourself in the Adobe quarterly report where they expect to get more money from "an increased demand for tooling products, server technologies, hosted services and applications". Now what happens if people disable flash, developers don't use it, and device manufacturers no longer want it? If YouTube even hints at HTML5 video I would short their stock.

*This has happened several times on Google tech talks that I have seen, it is amazing what people will say when they think they are only speaking to ten people and not the web.

Usable Linux on the laptop?

2008-09-11T23:54:00.006-04:00

Four years ago I picked up a used G4 apple laptop as a second machine. I kept OS X on it and happily used it. The reason I picked up an Apple was that for years I had tried to use Linux on a laptop, but never been able to very well. When I got the apple I discovered some 'amazing' properties that it had. At any point in time I could close the lid and it would go to sleep. If the battery got to low it would suspend to disk. Open up the lid and it would wake up and in just moments you are back working. Not only that but it would automatically find wifi and connect to it even if you were not in the same location. So what if the thread performance of the OS X kernel wasn't that fantastic, it worked really well with the hardware to give me the user a fantastic experience and I still got all my unix tools. Two years ago I picked up a macbook and to add to the fun I can stretch it to five hours on battery if I want to. Every once in a while someone would ask me why I don't run Linux and after telling them the above they try to convince me that it is much better now. I shrug my shoulders and say I am happy so I see no compelling reason to switch to Linux on my laptop (my desktop is Linux).

At my new job I got a laptop and was excited to find out it was getting a Lenovo, one of the laptops that had support for Linux, formally part of IBM and a good rugged laptop to boot. Putting the latest Ubuntu on there I figured was the perfect combination. Ubuntu had been talking about putting themselves on laptops so I assumed that it would all work. After having it for a week here is my notes:

100 Minutes - Not sure if this is Lenovo's fault or Linux, but I get 100 minutes off the battery with light usage. Compiling webkit or something big and continuous and the battery is shot. You plugin your laptop when it has 20% left and usually it starts out at 80% just from being carried to work or being in my bag. So if I am real lucky I get less then fifty minutes before it starts warning me to plug in laptop. You could say this is really a portable workstations, but my macbook also has dual core and 2GB ram and it gets much better. I am leaning to blame Linux because I couldn't believe a laptop manufacture would actually release a non-gamer laptop with only 1 1/2 hour battery life.

Maybe I have become spoiled, but when I close the lid I expect the laptop to go to sleep. Closing the lid on the Lenovo doesn't do anything other then turn off the screen. The first day I put it in my backpack only to find my backpack getting *very* hot by the time I to MIT and to top it off the computer running out of power before the next day loosing my state. With the screen off and the graphics hardware asleep I can't imagine what is causing the heat buildup with the desktop idle.

Suspend or hibernate. Ask anyone on the street to tell you the difference and they will tell you to go away because they don't know. Laptops shouldn't bother asking you. By default they should sleep and suspend only when the battery is near gone.

In the event that the laptop sleeps for a very long time I would expect that the shiny Lenovo would suspend to disk when the battery is very low to prevent me from loosing my state, but nope Linux is happy to loose it.

When I do get it to suspend I would expect that the resume would be snappy. The first few times I assumed it wasn't working and hard rebooted. Turns out it takes nearly ten seconds to resume under Linux. Resume from sleep on OS X is near instant, maybe half a second.

On the plus side wifi in Linux now works...

Pick the transformer

2008-08-20T13:32:00.005-04:00

I finally got around to making a fun little website where you are shown two classic Transformer toys from the 80's and you have to pick which one you get to keep and which one ends up getting traded away. The site is called called Pick the transformer. It keeps track of the votes and is able to rank all of the toys and show some basic stats on the votes. It is fun to play with and easy to find yourself clicking and clicking.

Back in 2000 a site called amihotornot.com (now hotornot.com) was launched. The site showed the viewer a photo of a girl and you got to rank her 1-10. While not a perfect system it was simple enough to get its internet moment of fame. A little while later another site called pickthehottie.com was put up that showed two images and you had to select which girl you found more attractive. Girls submitted their own image because they want to see where they rank and guys, well they want to look at girls. The sites have expanded sense then to have all sorts of options and extras, but at the core it is just trying to rank photos. The algorithms between the two sites are very different and believe that pickthehottie is much better then hotornot.com. On hotornot you are faced to rank a photo between 1-10 with no guidelines as to how to make judgments. While it is simple enough to say that photo X is near the top if most votes put it at a 10, you can't know with very good accuacy where the photo ranks in relation to the other photos that are mostly 10. (or another way who is the hottest?) This is further problematic when you consider that most photos will cluster around several numbers such as 1, 5, 7, and 10. To rank anything from 1-10 you need to first get a metal baseline of what are the two extremes of 1 and 10. The first few images you have to vote on are very speculative and it can't be until you have seen a number of images that you can consistently rank images. But even once you are consistent you must constantly try to compare the current image with all previous images you have seen which can cause a 7 to become a 9 or 5 over time as you get more and more input or just change your opinion. Contrasting this with pickthehottie I think that many of those problems go away. Given the choice between two images users only have to pick the one that they like more. This is usually so easy that users don't even think about why they think one photo is hotter, they just know it. It is only when faced with the choice between two very similar photos that the user would even have to make a hard choice. Behind the scene every photo contains two lists of photos that are hotter or cooler then the current image. Just based upon the percentage of the two votes you could generate a ranking for each photo. Taking it one step further you can fine tweak the ranking by only counting the votes that were places when comparing two photos of similar percentages. This was a fun little two day project that came together very quickly and is fun to use.

Three years in Europe with Trolltech

2008-08-15T07:53:00.048-04:00

After three years at Trolltech today was a sad day as it was my last day. When my wife and I originally decided to move to Norway we wanted to stay for three years and it is satisfying that we were able to reach that goal. The time here has been wonderful, sometimes stressful, but something I will always remember. I thought it would be worth going back and putting together a journal entry of projects I was involved in and photos I took.

In January 2005 My wife and I visited Oslo for an interview at Trolltech. The day before my interview my wife and I walked to the address to make sure I could find the building. There we discovered that Trolltech was located in a somewhat rundown building, but the next day when I went inside I found it to be much nicer and more importantly filled with interesting people working on captivating projects. I took the job and six months later we moved to Oslo. It turned out we arrived in Oslo on the day of the big Qt 4.0.0 party, unfortunately I didn't know about this and we stayed at the hotel. There was around thirty developers and Trolltech was still small enough that the computer at your desk had an actual internet IP.Being new to Trolltech and Qt4 I figured I would have been assigned a minor task to some rarely used bit of code, but after getting setup I was asked to add a feature to QWidget and my first commit was adding support for freedesktop.org startup notifications. After using Qt for eight years the idea of adding code, a feature nonless to QWidget was amazing.

The first three months we stayed at the Trolltech Markvien apartment which is just a ten minute walk from the office. When we flew to Norway we came with just four suitcases. There was no internet at the apartment and as my blogs from back then can attest to I took advantage of the Trolltech library and read through a number of the books. Every weekend for the first two months Jen and I went to one of the tourist attractions around Oslo because we knew that if we didn't do them then we wouldn't at all. We went to the ski jump, the art museum, viking museum, the fjord, the statue park, and many more places. Moving to Oslo we sold both our cars and were now walking everywhere. Along with visiting attractions we frequently would just go off walking around Oslo to see what we could find. There is a path next to a river that flows through Oslo with waterfalls that we would walk down to get to the center of town. It was a very different experience then we were used to back in the states, but the biggest shocker has to be women on bikes. It is sociably acceptable for every day adults to ride bikes, even a women in a skirt which was something you would never see in the states.

On the first day of work I asked Marius (another developer at Trolltech) where the grocery store was and he told me it was across the street on the way to the apartment. I was very skeptical of this as I had walked to work that morning and hadn't seen any grocery store. That evening on the way home I went into the place where he said it was to discover a store about the size of a seven eleven. This I was very quickly about to discover was the average size of a grocery store in Oslo. It has most everything you need, just not fifty choices of everything. Rather then one massive store per town there are many little stores all over that are within walking distance. It makes a lot of sense when you think about it, but coming from Super Stop-n-Shop's it was a bit of a culture shock. Of course everything tastes different too, but no doubt after living here for so long when I return I will fondly miss food that I could only get here.

A month after arriving I went to my first aKademy in Spain. We went for the full ten days and in typical American fashion Jen and I both brought a suitcase, while everyone else from Trolltech only had backpacks. A good lesson to learn for all of our future trips in Europe. As it was my first aKademy I finally got to meet everyone who I had interacted with online and on the mailing lists. I had a blast going to all of the presentations, discussing endless ideas and possibilities and even getting in a bit of hacking. And after many years I finally got a stuffed Konqi which has sat happily in my office. We made grand plans thinking KDE4 could be out in less then a year, but little did we realize how wrong we would be. For me post aKademy I spent a bunch of time cleaning up kdelibs and in particular reducing KApplication to the point where I was able to run a KDE application that used QApplication.

Just a few months later on December 19th, Qt 4.1.0 was released. The biggest addition for me was qtestlib. The super cool little test framework that had been part of Qt-addons was now part of Qt. During aKademy that year it was decided that including it in Qt would add a lot of value and reduce the duplicated effort that had begun within KDE to make their own test framework. I am very big on testing and ended up writing a tool to generate autotests stubs for you. All you had to do is pass in your class header and out comes a file just waiting for you to flesh out that tests your entire class. And when you think you are done writing tests you run it through valgrind and feed the callgrind file into the new code coverage tool I wrote to discover just how much of your class you didn't test. And for anyone making a custom QAbstractItemModel I wrote the Modeltest to help you make your model better and more bug free. One thing I always wanted to see was releasing the Qt autotests with Qt which might (unless it doesn't) happen later this year.

During this time Trolltech was growing and in December 2005 we moved to a brand new building with plenty of empty space for us to grow. Me and Marius traded a view of a waterfall for a view of a hill, both better then any office view I have had in the states. Over the next year the development team at Trolltech doubled as we expanded. The new office was big and much more in line with Trolltech becoming a company rather then a group of hackers, but that didn't stop us from having a paper airplane contest off our 6th floor walkway.

In June of 2006 Trolltech sponsered a cabin trip for the KDE Developers to hack on KDE4. Developers from all over the world came to a little town in Norway and spend a week hacking in a very big cabin. Since the last aKademy KDE4 had deteriorated and was in need of some work. The first two days were spent fixing up KDE left and right and finally It was there for the first time that I was able to see and run KDE4. It was also there that QtWebKit was born. It was amazing to watch as the guys discussed the future of KHTML, grabbed the WebKit source and hacked on it until at 2am on Zack's screen there was the Google homepage. One of those moments you can't plan for or buy tickets to. Meanwhile I was fixing something in the KDE about dialog, depressed at the comparison I went to bed. While there the world cup was going on, Trolltech IPO'd, and the various outdoor meetings we had were just fantastic. For me personally this cabin trip ranks at the top of my entire time in Europe.

A few short months later Qt 4.2.0 was released on October 10th 2006. That date had been set in stone in time for dev days and after the fact everyone agrees that it was pushed out too early, a mistake we tried not to repeat since, pushing back releases by a few weeks when we knew that the quality would be significantly better. Unsurprisingly Qt 4.2.1 followed very shortly after. At some point after 4.1.0 (I don't recall when exactly) Mathias gave one of his fantastic speeches which determined one of the big directions of 4.2 which was better integration with the desktops. For my part I wrote QDesktopServices, but there were many more such as QCleanlooksStyle, QNetworkInterface, QSystemTrayIcon, QKeySequence's standard shortcuts, integration with Glib, and more. Another new class I got involved with (and wrote a few older versions of) was QTimeLine, something I have had lots of fun using ever since it was created. 4.2 also saw that introduction of the eye candy changelog.

Around this time I got involved with the netflix prize and ended up making a framework that used Qt and included a little app that you could use the view the entire database. While making the tool I improved Qt so that QTableView could display all 101 million rows of data. I didn't win the prize, but I had a lot of fun, learned a lot and if I ever get access to a farm of computers know exactly what I am going to mess around with. And to top it off every few months in the mail I get a thank you book or dvd from my amazon wishlist which is very cool.

For the first time since moving to Oslo Jen and I returned to the states for Christmas. After living in Norway for a year and a half we could both speak very bad Norwegian. With no real Norwegians around we happily talked to each other using our horrible pronunciations and bad sentences structure. We could understand each other and we were happy. While in the states I got a MacBook and have had it automatically take two or three photos every day for the past two years. Looking through the photos there are many photos of people, books, and other images of every day life that I would normally not have taken a photos of. It is nice to have captures the memories that would have been lost.

Returning to Oslo I formed a group and wrote up a proposal to get a Wii for Trolltech. A month or so later a box arrived with a Wii and extra controllers which was promptly setup (The Wii being somewhat easier to obtain in Europe we only had to wait a month). We have had tennis tournaments and many after lunch Bomberman93 (an absolutely fantastic five person, ten minute, easy to learn, multiplayer game) sessions ever since.

Looking back I was surprised to discover that I have been using Git for over a year and a half now. One of the first things I used it for was to port an annoyingly addictive Qt2.x game to Qt4.x called Anigma. The big thing in my life was that I cut my hair which had been very long. The first few weeks were very freaky and I kept reaching back to move it out of the way or to play with my phantom hair.

On my creative Fridays I had been hacking on a little class called QColumnView. A nice little class that I am proud of that got into Qt 4.3 which was released June 1st 2007. Also in 4.3 I wrote a new QFileDialog. The majority of the work was in the model, but I also tried out a new look a feel for the dialog that was a hybrid of OS X's and Vistas with animations and other 'improvements'. Which everyone completely hated and got torn to shreds online so I reverted the interface back to the 4.2 one. The biggest thing for me in the 4.3 release was QtScript. The kick ass, fast scripting engine that I had been messing around with for several months and used to play around with genetic algorithms. I even wrote a QScript profiling tool. Overall Qt 4.3 was a good solid release with more improvements then features I would say.

In the fall of 2007 I finally sat down and started to learn Lisp something I have always wanted to do, but just never made the time to. Unfortunately a much more exciting project was just around the corner that took my free time away...

Shortly before Christmas while at lunch one day I had a conversation with Simon and Lars about the QtWebKit module which had just been merged into Qt. I had been asking about if there was plans to make a little browser that could be used to help test the module. In WebKit there is the QtLauncher, but it was more for developers and not something that can be used for day to day browsing. They were also interested in such an application and I offered to help make one. All through Christmas I hacked and my last blog entry of 2007 gave a hint about what I was working on and what would become the demo browser. Starting with the cookie jar I took each feature one by one, fleshing them out and only integrating them when there was a manual test and matching autotests. I spent all of my free time hacking on the browser and then part of my time at work tracking down and fixing bugs in QtWebKit and the new networking code. I was ruthless with any segfault I found and by the time 4.4.0 was released QtWebKit was pretty stable. One of the very first QtWebKit bugs I fixed was adding support for tooltips so xkcd's tooltip would work. Unlike FireFox2 which would cut off long tooltips QtWebKit tooltips could wordwrap. Just like my first commit to QWidget in Qt the feeling of committing a patch to WebKit was really cool. In February Trolltech had its annual ski trip in the mountains. Before we left I printed out all of the code for the demo browser and with the help of other developers did a complete code review on the bus and in the cabins. This helped improve the code for the announcement of the little project. One day I was asked if I would be interested in writing a QtWebKit article for Linux Journal. Having never been published in a magazine before I jumped at the option and a few months later got three Linux Journal magazines in the mail which was really neat to see an article I wrote in print. Shortly before Qt 4.4.0 was released I forked the demo browser and created Arora. Although many people had been interested in seeing it forked much sooner I didn't want to do that until 4.4.0 was released to get as much feedback from everyone trying out the Qt 4.4.0 beta.

Around this time it was announced that Nokia was going to buy Trolltech. For me the biggest bummer had to be when I saw them taking down the Trolltech sign off the building the night before we became Nokia. Overall Nokia has not caused any major changes of doom as some feared and I expect Qt will continue to improve, grow and be even more open in the future. Trolltech has a lot of smart people in it and they wont let anything happen to Qt.

On May 2nd Qt 4.4.0 was released. This was while I was in Italy and as Qt got lots of press the demo browser also got mentioned various places (with lots of questions ending in my inbox), but I was unaware of all of this until I got back as I didn't think the 4.4.0 packages were going to be released until the end of the month. I had a few stress filled days catching up before things returned to normal. Other then the demo browser I had also gotten in QSystemSemaphore, QSharedMemory, QLocalServer & QLocalSocker, and the QFileSystemModel. But my favorite addition has to be QFile::map, something I have implemented several times on personal projects. Qt 4.4.0 also had a number of brand new features and I think that this release was the best .0 that Trolltech has had to date. One feature of Qt 4.4.0 that I think is a hidden gem is QtConcurrent. I have used QtConcurrent in some personal projects since it was first announced and the ease that it lets me use both of my cores to double my speed still amazes me. With the possibility of having six and eight cores very soon the speedup in performance will not be insignificant.

I am leaving before 4.5 is released, but I have been able to squeeze in some stuff including the new QTabBar and QTabWidget improvements and implementing a cache for the http backend in QNetworkAccessManager. I am looking forward to the 4.5 release.

Around this time Jen and I knew that we would be returning to the states and so we made trips to Italy and the camping trip to Trolltunga. While living in Europe I got the opportunity to visit Spain, Germany, France, Ireland, Scotland, England, Italy, and Jen visited even more. Because they were spaced out we took the time in each place rather then rushing from one to the next during a month long Europe vacation. I have tried to limit my blog to technical related articles, while Jen's blog contains many more of our every day adventures from visiting cities to laughing about the food we would find.

While visiting Euope can be a lot of fun, the act of living here has open my eyes to all sorts of things. The idea that we could get by without a car is one we could not imagine three years ago and yet somehow it works out just fine. With so many people from work walking to get there is is no surprise that on an evening stroll or walk downtown that you will run into someone you know. The closeness of the community is very nice. Many of the guys are Trolltech have stories of when they were in Army, unlike in the states where you get to sign up, here they don't really have a choice and although I knew that some countries did that it didn't really hit me about what it means until talking to many of them.
Another difference was the banking system where in Norway it is far ahead of the states. No one has a checkbook and my bank didn't even accept checks, all your bills are paid online, and if you want to send money to a friend it is easy to do online. Even taxes are calculated for you by the government and if there are no errors you can send a SMS to confirm the form and you are done.
The metric system, 24 hour clocks, A4 paper, currencies, and other things you don't think about when visiting, but you have to deal with when moving were just a few other things we had to adjust to. At least with the 24 hour clock I plan on continuing to use that and getting a 24 hour alarm clock in the states so I will never set the alarm for 7pm when I meant 7am. Overall living in Europe has been a fantastic once in a lifetime experience and I am glad that I did it.

While at Trolltech they managed to put out consistent releases with new features and improvements. From the time I started to my last day they strive to be open, from making the Windows version of Qt GPL, the launch of labs.trolltech.com, creative Friday's and involvement in many open source projects such as webkit, git and KDE, they have risen the bar about what a company can do. But none of this would have been possible without the people behind the scene. Above everything else coming to Europe and Trolltech has been about the people I have met and worked with. Trolltech is filled with some of the smartest and most interesting folks I have met who enjoy what they do and it shows. They are friendly, welcome you into your home and will help you out when you are completely confused.

Even though I am leaving Trolltech and returning to the U.S. I plan on continuing to use Qt, and probably submit a patch or two. Arora has really taken off and has an active set of users and developers and I am really looking forward to what it will become. Hopefully I will return to Europe to attend aKademy and get to see everyone again.

And to wrap it all up I thought it would be fitting to have the Qt4 dance.

Code Review Should Be Free

2008-08-07T14:52:00.006-04:00

Depending on the definition "code review" can mean a wide variety of things such as formal code review or automated code analysis. For this article I am talking about when a developer has a patch to some code base and would like someone else to review it before it goes into the main repository.

Just the thought of having a code review will frequently cause developers to spend extra time making sure a patch is as good as it can be. Code reviews typically catch bugs, design flaws, missed edge cases, inconsistent or confusing code style, and more.

There are several core ideas for code reviews, the more of them that are followed the more successful the technique will be.
- Review of the code by one or more persons before committing it to the central repository.
- Record comments about a patch so suggestions are not forgotten as a patch changes.
- Patches should be easy to review and test.
- The developer shouldn't feel like the code review is holding them back or that they are waiting for a review before they can continue working.
- Reviews should integrate easily with the existing work flow.

With only these five requirements it shouldn't be too hard to do. So what are the most common ways that developers do code reviews?

Email pass-around

The most common form of code review I have seen is the email mailing-list of commits. It is so common that even when you find any other technique you will often still find this one running too. This is mostly because it is "the example" of how to add a hook to your revision control system and usually extremely easy to setup if not done for free. It works by having a hook in the primary code repository that sends an e-mail to a mailing-list with the diff of every commit. This of course fails the first and most important requirement which is to get the review before committing. The idea is that everyone reads commits they are interested in and replies to the mailing-list when they spot something wrong which another commit fixes. In practice most commits are not reviewed and most people read only a few random commits before marking them all as read. This of course only gets worse as the project grows and the volume of mail on this mailing-list increases to the point where no one reviews. To top it off when you see some code that works, but has a few confusing variable names you wont say anything. In fact it takes quite a bit for anyone to actually hit the reply and comment on a patch; code that doesn't compile, possible security bug, introduction of bad public API or some other major and obvious bug. And this works both ways, the committers know that they can get away with it so they don't make their commits as polished as they could.

Cons: By the time the e-mail goes out to the mailing list the patch is already committed into the repository. Once the code is in the repository it is much harder to ask someone to make smaller fixes and can even be taken as an insults. In practice there is an extremely low volume of actual reviews that are done.

Pros: Having nothing to do with code review this method is often best to occasionally let you see what is going on elsewhere in the project. In fact it has spawn a whole sub world that digest commits and write up reports such as the KDE commit digest.

Over the shoulder

Less common in the open source world (and in the corporate world) is the in person code review. This is where you have someone physically next to you while you explain the patch. While better it has its own drawbacks. When you go over to do a code review for someone else they have usually been sitting around for five minutes waiting for a code review and quickly walk you through the code. Because you are face to face, you can often ask any question and quickly determine if the developer had explored all the options you can think of. You can ask if the unit tests have run, if there is an example, etc. At the end of the day the results vary wildly with some reviews being in depth while others being just a quick glance. When this is the primary code review method and the person that would normally review some code is not available (vacation, sick, out of office) the commit might go in without any review.

Pros: Face to face reviews can be very quick and I can quickly turn into pair programming. The process of explaining the problem and patch can often times cause the developer to think of edge cases themselves to test. Unrelated this is one way to teach someone about a piece of code.

Cons: Many times the reviewer can feel rushed as they try to digest code that the developer quickly explained. The developers tells the reviewer the patch is correct even before the reviewer has reviewed the code. The developer who is best suited to review the patch might not be around and a developer who does not know the code and is not as able to properly review the code is drafted into doing it.

E-Mail diff

Bits of all of the above, a diff is passed around through e-mail which can spawn a thread of comments. The biggest downside is that there is a higher cost to entry and people are less likely to send smaller or simpler patches via e-mail. Often times the only patches that are sent via e-mail are larger ones that go through a number of revisions, and yet the code being e-mailed around isn't in a revision control system loosing out on the ability to see the diff between versions. Because the developer resends the entire patch every time they make a change the more revisions that are sent in a thread the less actual reviewing will be done.

PasteBin

In the past few years I have seen the rise of paste websites such as http://paste.lisp.org/ that provide a simple way to paste some text. Usually no login is required and it takes only seconds to do. Developers will generate a diff and paste it onto these sites and then ask for a review over IRC or another chat system. There are even command line tools to make pasting code straight onto the website even easier such as 'svn diff | lpaste'. Pasting while cheap and easy has drawbacks such as requiring the other person to be there right then and the paste is often missing full context such as what was just above a diff. Because pasting is usually done in conjunction with chat the paste sites don't provide a way to add comments to a patch.

Pros: Very easy to do. Can be done with someone that is remote. As it is just text it is not constrained by any system or process. Also used sometimes for a quick way to send patches or other files rather then e-mail. Because you are also chatting via IRC you know that patch is being reviewed right there and then vs e-mail which could be a week.

Cons: The patch is frequently only read and almost never applied to a working repository to test even if it even compiles. This is just another tool that has absolutely no integration with any other tools you use such as the source repository. Feedback is always done via another format such as IRC. Requires live interaction between both parties and requires the manually pushing of the diff up to the website in the first place to get a review. Many paste sites have an expiration of a month and sometimes developers set it to one day causing frustration the next day when the reviewer tries to access it.

WebTool

In the past year or two several code review website tools have popped up. http://www.review-board.org/ and Rietveld are the one that had obtained some blog press. The review tools all try to improve upon the paste idea by managing patches. You can add new patches, request reviews from developers, add comments to patches, and update the patch. When I first heard of the tool I though it was fantastic and played around with it, but ended up not using it for very long. Fundamentally it requires a lot of work on the part of the developer. For every patch there is a lot of manual work that needs to be done. Even when the patch is approved you still have to submit the patch and then mark it as committed in the tool. Not to mention you need to have yet another way to watch for updates to your patches and for patches you need to review. When testing out the systems I only ever used one patch, but in production I had many patches, some of them on the same file. This caused problems as the patches were approved and other patches got stale and the tool would get confused. Fundamentally a review system needs to have the foundations of a revision control system to properly work and be closely tied into the revision control system that is used. Most of the review sites out there are little more then paste projects that got scaled up, they will let you down. To further illustrate this: you can patch your patches. This causes the review system to keep what is a branch on top of the current tree that you can view. This eventually will break by not behave like your real revision control system does. The most common one I saw was where I wrote patch A, someone else submitted B and then I updated A and uploaded it. Rietveld would freak out and break. And because the review systems are so big they eventually go down for a day here or there. All of your patches are up on the system when the review system goes down you can't do anything either. Just as pasting got popular because it was so very easy, a web tool probably wont get very popular because it is just to much work.

Is there any better options?

So after that whirlwind tour of what is currently in use out there I want to present something I have been using the past few months that takes advantages of distributed revision control and a little known feature of GitHub.

One behavior I have witnessed is that once a developers pastes a patch they wont do anything until they get a review. They won't commit until they get a review and they don't want to work on the next task until they commit the first. Nice catch-22. Using a distributed revision control system such as Git (or any system where branches are cheap) a developer can commit to the branch, post the patch for review, and continuing hacking in another branch giving the reviewer(s) as much time as they want to review the code. Patches sitting in their own branch can be worked on and revises for as long as needed without feeling as pressured to merge it 'right now' as often happens when a system does not use branches.

So just switching to Git solves one annoying code review problem, but can it do more? Several months ago I started using GitHub for hosting Arora. Every developer has a fork of the main repository (including me). When a branch (often just one change) is ready it is pushed up to GitHub and a pull request is made. When I get a pull request I can view the patch which is the same as e-mail pass-around and paster, but because the patch is in the source repository I can do all of the following very cheaply:

- Review the patch in the context of the entire file. I can glance at just a patch, view the entire file, its history, other branches, and well anything I want because I have the full repository right there.
- Review the changelog entry to make sure it is following the project format, spelling, etc.
- Checkout the code and try it myself. The number of times I have actually downloaded and applied a patch from a paster is very small, maybe twice a year. But with Arora almost every time I have fetched the repository, built the branch, played with the new feature / bug fix, ran the auto tests, etc. It is *very* cheap and easy. When I get a pull request from user foo, I run git fetch foo to grab their changes and because the patch is in its own branch compiling and running the patch is as easy as checking out their branch. No patching required.

Another nice aspect is that when viewing the change on GitHub you can click on any line number and add a comment right there. Add as many comments as you want to the different issues you spot. When the developer commits the next version you don't have to review the whole patch, you only have to review the patch to the patch to make sure that what you commented on was fixed. Because the branch is a real Git branch and not some files on a website review tool it understands patches to patches and when upstream is rebased just fine and never has issues.

Unlike review-board there is no extra work involved to review code. I get code review features for free with doing my normal everyday committing and pushing. When I get a pull request that is perfectly fine I can just merge it and reply back with a "Thanks!". But if I want to go in-depth with a review and leave comments and have them upload a new versions I can. If I want multiple reviews of one of my changes I can easily get that with multiple pull requests (with a comment just to review and not merge). To get reviews of my code I already push up my branches so it is nothing more then going to github and clicking on the pull request button.

Reviewing with GitHub isn't perfect. I would like to have an RSS of all the comments I have left and another one of all the comments others have left on my repository. Currently they are simply placed in your news feed. But what is in place today works well enough and is far beyond in my opinion the other code review options.

With distributed revision control systems and sites like GitHub or Gitorious code review is such a simple add-on to already proven workflows, it's essentially free.

Photo by Sebastian Bergmann under the CC license.

An adventure to Trolltunga

2008-07-30T15:54:00.023-04:00

A little over a year ago on reddit I saw a picture of a guy sitting on the edge of a rock. Not just any rock, but a clip of a rock that jets out of a vertical side of a mountain with a 350 meter drop. In the comments I discovered that the photo was of Trolltunga and is right near by in Odda Norway. Along with some other trolls from Trolltech we rented a car and drove to Odda for a little camping trip. The drive took around five hours, but we were in for a treat as the two lane highway though Norway is filled with fantastic scenery. The highway went through many of the mountains and once there was even a corkscrew tunnel. About half way through the trip you come out of a tunnel up in the top of the mountains dotted with snow. We stopped there and had a snack and took photos before continuing on.

To go with the mountains were numerous waterfalls.

Once we reached Odda we started our journey at the funicular railway which takes you up the side of the mountain which is 960 meters high with a 42 degree angle slope. According to our information the train was not running this summer and we would have to take a very steep path up the side of the mountain with full backpacks of camping gear. This exhausting start to our trip took a little over two hours.

But is was rewarding to reach the top and after a short rest ...

We continued up the path for another hour and finally made camp at a small clearing by a stream. Making dinner we had an absolutely fantastic view of the sunset over the snow topped mountains.

Being so far north the sun was only set for a few hours and in the morning after packing up camp we started off the trip to Trolltunga. On the way:

We walked through snow ...

We saw crystal clear ponds formed from melting snow ...

Saw many streams and waterfalls of every shape and size ...

and walked along the edge of the fjord.

After hiking through the mountains for about four hours we arrived at Trolltunga!

After taking many pictures and having lunch we headed back the way we came. Along the way we met someone who told us that the railway was in fact running, but only twice a day, at ten and six. We hurried back and four hours later arrived just in time for a ride down the insanely steep mountain. Riding in the front of the car it was like a roller coster ride where you go off the edge, except it took ten minutes to get down.

In all we walked around twelve kilometers over the two days. Originally this trip was all about seeing one little site, but it turned into a full adventure with all sorts of things to see at every step of the way. If you every get the opportunity to goto Trolltunga I would highly recommend it and it is without a doubt one of the highlight of my time in Europe.

Update: After submitting this to reddit and getting a bunch of votes I got a reddit header.

Upgrading the iBook hard drive

2008-07-12T17:25:00.003-04:00

A few weeks ago I picked up some new laptop hard drives to replace both the one in my MacBook and the one in my wife's iBook. The MacBook's hard drive upgrade process is a very easy five minutes process.

Pop out the battery

Unscrew cover

Pull out hard drive

Meanwhile the process of replacing the hard drive in a iBook's is a three hour hundred step process involving tons of little screws, a technical manual and taking apart layer after layer of the laptop. When the laptop was fully taken apart and spread across my entire living room I snapped a few photos of the geek porn.

Puzzles, Algorithms, and Qt::Concurrent

2008-07-12T07:40:00.006-04:00

ITA software, a very cool Lisp company has for years put programming puzzles up on their website. Although the puzzles are fore hiring, they are still interesting and every once in a while I take a stab at one of them. One that recently caught my eye was this Sling Blade Runner puzzle. Although it is not in the archive of retired puzzles the problem itself is NP so I am not worried about showing off my answer as everyones solution will no doubt be different. I myself wrote several different solutions including a genetic algorithm and it was turning out fantastic results, but it was taking days to run v.s. the solution I present here which takes only minutes. Onto the actual problem:

"How long a chain of overlapping movie titles, like Sling Blade Runner, can you find?"

Use the following listing of movie titles: MOVIES.LST. Multi-word overlaps, as in "License to Kill a Mockingbird," are allowed. The same title may not be used more than once in a solution. Heuristic solutions that may not always produce the greatest number of titles will be accepted: seek a reasonable tradeoff of efficiency and optimality.

The problem description itself is a little vague leading to different goals depending upon how you read it. I choose to count the length by the number of movie titles that were used rather then words or character.

The first part of the problem is just reading in the data (the data file has its own issues for you to solve). Just from googling you will find several blogs where the discussion is mostly all about just reading in the data and running a depth search. This finds chain lengths of around 220 very quickly and while nice doesn't attack the real problem of the algorithm.

Once you read in the data at the core you end up with a list of nodes. Each node contains a list of nodes that it points to (and optionally a list of nodes that point to it). In other words your challenge if you accept it is to find the longest path in a directed graph.

My solution is built upon two algorithms. The first algorithm takes a starting node and produces a list of long paths. This algorithm wont guarantee the return of the longest path for a source node, but it will return very long paths in a very short amount of time. The pseudo-code version of the first algorithm in (called findLongChain in the code):


   set1 is populated with a source node
   while (set1 is not empty)
       for each node in set1 all paths are followed.
           If the path to the next node is more then the
           currently known length to that node it is
           stored and that node is added to set2.
       set1 = set2

Using just this algorithm a respectable result can be found.

These long paths can be sent through a second algorithm which attempts to increase a chain length by finding local longer paths using a limited depth search. The pseudo-code version of the second algorithm (called expandChain in the code):


   for (i = 0; i < chain.length; ++i)
       Starting at i do a depth first search with a max depth
       of X where X is a relatively small value to find a longer
       path then the current chain[i] -> chain[i + X]. If a longer
       path or subpath is found replace it in the chain.

Once the graph is constructed each node is passed to findLongChain which runs its best chain through expandChain before returning it. Once each node has been run through findLongChain the root node of the best chain is run through findLongChain, but this time all the best paths are run through expandChain to see if there is anything longer to find.

Because findLongChain is completely stand alone I was able to take code that was originally something like this:


    Chain longChain;
    for (int i = 0; i < todo.count(); ++i) {
        Chain chain = findLongChain(todo[i]);
        if (chain.count() > longChain.count())
            longChain = chain;
    }

and change it to use QtConcurrent's mapped reduce function to take fully advantage of my dual and quad core computers. The ease that QtConcurrent lets you scale your solution is fantastic.


    Chain longChain = QtConcurrent::mappedReduced(todo, findLongChain, reduce);

While hacking on my solution I came across Jeremy Faller's website who also had been looking at the problem and was able to produce good results. (our approaches were completely different and we didn't see each others code until after we had finished). I took his longest length chain of 318 and ran it through my expandChain function to get a chain with length 326. A very respectable result found by the combination of our two programs.

The source code for my solution can be found on github in my sling blade runner repository.

After compiling run './sbr' and in around ten minutes on a quad core (subsequent runs last around five minutes). It will print out the best solution that was found which unless you change the code was a length of 312. If you uncomment the DEBUG define at the top of main.cpp the progress through the nodes will be printed as well as the best chain as it is found.

Hacking on this problem has been a lot of fun. Because there is no right solution I was free to try out all sorts of different approaches and spend time reading up on papers on graphs (the ones I didn't have to pay for). Spent some time messing around with Qt::concurrent and how fast I could get it to run. While working on the problem it was fun and motivating as I was able to pass everyone else's (the ones I found via Google) longest chain. The best score found by my code is 313, but I am more happy about the combined length of 326.

Cast off

2008-07-02T17:10:00.009-04:00

After three weeks I got the cast off my broken hand on Friday and I can use both hands again. It has been an incredibly interesting experience. I took down notes of events throughout the three weeks that were interesting so I could write this blog entry once I could type again.

The word that can best describe the three weeks was "Slow". Everything, and I mean everything took twice to ten times as long to do. Being that I injured my primary hand my left hand was forced into service. I was surprised how quickly I was able to learn how to use my left hand to grasp and manipulate objects. After only a week and a half I was able to manage (not great, but doable) using a fork with my left hand, using scissors, brushing my teeth, and some basic writing. I found that the best way to write with my left hand was mirrored and in reverse which while cool becomes very cryptic if you are sloppy and I had to use a mirror a few times. I also very quickly learned how to use a right handed mouse in my left hand, in the same fashion mirroring the events that my hand should perform, not swapping the buttons.

Meanwhile with my right hand it felt like I was learning how to use it from scratch. After I got my cast even though I had three fingers "free", moving them or really doing anything would result in a lot of pain, and pain for the next five minutes at least. I very quickly learned not to use them. My first accomplishment note for my right hand was the ability to take off my glasses. The first few day my list was little things, but I soon I was able to master buttons and after a week I had acquired the "strength" to pull off my wedding ring on my other hand which is a bit loose to begin with. As the swelling died down I was able to bend the three fingers and finally start using them to type. The lack of strength in my right hand was something that would frustrate the hell out of me all day long as I would crash into my new limitation. I could place my hand on the object, but couldn't actually do anything. From not being able to open windows, doors, eating, typing, opening any packaging, to tying my shoes (My wonderfully supportive wife did that that the first week until I found out how to tie them with one hand) it was slow going. I had to re-learn and re-acquire the strength to do everything and anything with my right hand.

On the computer I very quickly learned a ton of application shortcuts and my bash profile now has dozens of two and three letter aliases (using mostly keys hit with my left hand). The first week or so I had very few compile errors as anything I wanted to code would take forever to type and I would sit there in my head running through the code I wanted to type at 100 miles an hour as I touch typed with my left hand and ten words a minute. Very quickly I learned to sit and think about any problem and explore as much of it in my head before coding anything because coding was just taking ten times as long. A real big plus was that I have a Kinesis Contoured keyboard. On the Kinesis keyboard the return key is on your right hand thumb and not your right hand pinky. After I got my cast off it took another three days before I could even get back the strength to push down the enter key with my pinky on my MacBook keyboard.

As you can see in the photos I got some art on my cast. With my new found situation of not being able to do much I found myself having the ability to spend lots of time drawing on my cast. All the art was drawn with my left hand. The first one I drew was the tail that wraps around and goes into my fist the first night I had the cast. Then a day or two later came the green raptor near my fingers. This was when I had to keep my hand above my heart to prevent lots-o-pain so it was drawn in that location so people could see it. The second one (week 2) I drew was the baby velociraptor scratching its head with its foot. This was when I was holding my hand mostly at a 90 degree angle across my chest so it was facing out. The second one is much better then the first as I understood the medium better. The second one was also drawn upside down from my perspective so everyone else could see it right side up. The last one I drew on the third week was the T-rex and this was when I had my arm at my side most of the time and so it was facing out. This one is definitely better then the other two.

Here is a photo shortly after I got my cast off. The whole right side of my hand is very swollen and green. Lots of pain and aggravation still, but at least the cast is off. An interesting aspect of breaking a bone in my right hand was that my fingernails on my right hand seemed to stop growing while I have cut my left hand fingernails twice. Not something the doctor tells you about. Almost a week after the cast came off and the green is still with me (not as much) so at this rate I figured that will stick around for a month or so and I was told I have three months until it fully heals. On my palm it is a big green circle in the middle which when pressed hurts like hell. Yesterday I tried to kill a bug that was flying around with my hands... it was a really stupid idea. As you can see I can type again and life for the most part is returning to normal. While setting the bone shifted so I no longer have a knuckle and my pink angles slightly to my ring finger, but nothing serious, really just cosmetic.

This experience has been a very interesting one. I was surprised just how much I was able to do with my left hand when given no other option and the process of relearning how to use my right hand was also a bit of an eye opener as I struggled to do the most basic tasks. But this is definitely something I hope never to repeat.

Linux Jounal

2008-06-12T12:33:00.004-04:00

Today I received my copy of the July issue of Linux Journal. Normally not blog worthy, but in this issue there is an article I wrote called "Using WebKit in Your Desktop Application" which walks through writing a small application with Qt and WebKit. This is the first time I have had an article published in a magazine and is very exciting.

Broken hand

2008-06-07T16:23:00.009-04:00

Broke my hand this evening and spent the rest of the day at the hospital, but I got the x-ray's on a CD which is cool. While messing around I hit the wall/door frame and decided to go to the hospital a minute later when I noticed my knuckle was missing (not to mention the pain). I broke the bone attached to my pinky and the front part went down and slid under the back part. At the hospital they pulled the finger back out put a cast on my hand. So for the next three weeks I am down to only eight fingers. The best part was my irc message on #arora

[5:08pm] icefox_home: broke wrist
[5:08pm] You left the chat by being disconnected from the server.

Boy does it take a long time to type with one hand.

Screenie

2008-06-02T11:15:00.000-04:00

Ariya Hidayat has put up a very cool little Qt4 tool called Screenie that helps you create cool screenshots of your applications for your website. Here is some screenshots of Arora made by screenie:

Italy

2008-05-15T16:29:00.012-04:00

My wife and I spent the past two weeks in Italy for our wedding anniversary. Flying into Rome we stayed in a nice bed and breakfast that was near to the main train station. Flying from Oslo where it is still a little chilly, Italy was absolutely perfect (minus a tiny bit of rain two of the days). Beyond visiting Rome we took some early morning trains out of the city and as a result saw a nice sunrise or two.

The first day trip we took was out to Florance where I encountered more people speaking English in public (in a country whose primary language isn't English) then at any time while here in Europe. One of the more famous paintings that we were going to see in Florence was "The Birth of Venus" which is currently in the Uffizi Gallery. But the experience at Uffizi was the worst out of all of the many galleries we visited in Italy. To start off they force you to ditch any water you have when you enter the building and only two minutes later they have you put your bags in storage. Later on we find it was so you will go to the cafeteria and buy nice overpriced water and food. Then we get to the painting and find it covered with a sheet of glass, but the worst part was that when you are standing at the perfect viewing spot (i.e. a few feet away in front of it) the top of half of the painting is covered by the reflection of the air vents from the other side of the room. Uggg. But beyond that Florence was beautiful and we spent the day seeing the sites and wandering around. I had never realized how tall the statue of David was and the accademia gallery did a fantastic job of displaying it.

The day after Florance we took a train down to Naples and visited Pompeii. There we had an exhausting, but fun day wondering around all of the ruins. Even though a lot of it was blocked off we could have easily spent another day there exploring. Later in our visit we returned to Naples to climb to the top of Mt Vesuvius. On the way up it was very cloudy and cool, but after walking around the rim the clouds finally lifted and we got some spectacular views.

I have come the conclusion that when available trains are a vastly superior form of travel over planes. We would wake up, get some breakfast, walk to the station, and hop on the next train to our destination. Big chairs, power plugs, huge windows, no hassle. There was absolutely no long stupid security checkpoints, no waiting half the day away, no feeling like a criminal when you haven't done anything wrong, and no waiting for your lost luggage. The trains always arrive in the middle of the city rather then 30 miles outside of where you want to be. They left on time and arrived on time. You don't have to buy a ticket a year in advance to get a sane price. Sometimes we would just hop on the next train not caring if it was the 10:00 or the 10:30 because you can. Another time when returning from Pisa we decided to take a completely different route through a different city and got a ticket for the new train in five minutes and quickly ran over and boarded the train moments before it left. Maybe this is what plane travel was like fifty years ago? I'm not saying I hate to fly, just that Trains (in Italy at least) seem to have it down really well.

Back in Rome there were many different things to see. Vadicant City, Four Fountains, Panthenon, Trevi fountain, Spanish Steps, Borghese Gallery, Coliseum, many different churches and much more. Trying to not walk the same way twice we ended up all over Rome and saw many unique shops. The biggest tourist trap was the Coliseum where I ended up having more fun photographing the people rather then the Coliseum itself. Throughout the day newlywed's appeared up to have their some photos taken with the Coliseum in the background which was an interesting local custom which I can't seem to find any reference to on Google.

Our final day trip was up to Pisa where we had some ice cream and took the tour up to the top of the tower. Pisa was a good example of the "little European town" that is in so many of the american movies show as the perfect place to live if you could.

Returning back to Oslo it was Rainy and cold. But not to worry, the days are long and I know it will be a beautiful summer as always in Norway.

Of course while I was away the GPL version of Qt 4.4 was released and got press everywhere. With that Arora got mentioned a number of places and I came home to find lots of patches and questions. I'll be going through them shortly and will try to get caught up as soon as possible.

Demo browser update

2008-04-05T16:34:00.014-04:00

As Thiago has recently blogged about the Qt 4.4 release candidate is now out. Since beta1 I have been spending a most my weekends hacking on the demo browser and there are many improvements in this release.

Bookmarks
The first new feature is bookmarks. Using the same open standard format at Konqueror, I have two little classes that can read and write the XBel format. These class use the QXmlStreamReader and QXmlStreamWriter so they are super fast. Similarly to History there is a base bookmarks class and on top of it a model that is used in the menu, bookmarks dialog, add bookmark dialog, and in the bookmark toolbar. Lastly there is an import/export so you import Konqueror's bookmarks. I tried to choose a suitable set of simple default bookmarks. Getting bookmarks was a nice milestone. With my bookmarks in the demo I found myself using the application more and more.

ExLineEdit
The QLineEdit that contains the url got an upgrade. It is now its own class UrlLineEdit which is a subclass of another new class ExLineEdit. ExLineEdit was made to make it look like a QLineEdit with several widgets inside of it. For the url line edit I wanted to be able to put a widget on the left that displays that current url icon and has dnd support. The coolest feature of the UrlLineEdit has to be the progressbar like loading of the background (see the screenshot). When it doesn't have focus it will fill with a gradient to show the current page's loading progress. When the url scheme is https it will also have the background color be yellow. For those who are using the Oxygen style, note that there are some painting errors because QStyle::SE_LineEditContents is either not implemented or wrong (i.e. I haven't bothered to actually look into the Oxygen style's code, but all the other styles return the same rect inside the frame which cause me to suspect it has a bug).

Private Browsing
Safari has a mode called "Private Browsing". This mode essentially doesn't recorded history, icons, accept cookies or many other things. Some call it the porn mode and there are many similar extensions for Firefox. This feature is built into WebKit so QtWebKit has a matching attribute you can set. It was only an evening of hacking to add private browsing modes to the demo browser classes and now it too has this feature. Currently it is very safe, even maybe to the point of being too safe. When in this mode sites can neither set nor get cookies. Perhaps a temporary cookie jar should installed when in this mode.

Settings
A handful of new settings, including letting you specify the download location, what should happen when another application asks to open a link (new tab or new window), the ability to turn on/off javascript and plugins, proxy configuration, and the ability to set a user stylesheet (i.e. poor mans adblock).

QtWebKit
Using the browser every day for the last few months in both OS X and Linux I found a handful of sites that didn't work for one reason or another. Most of them have been fixed. A few http issues, some Qt bugs, and issues in the QtWebKit port. With cookies set to always accept, GMail and Google's login is now working constantly for me. Getting better every day.

Overall application improvements

The ability to print and a print preview dialog

Added SaveAs Action

The Windows menu contains all of the top level windows.

Many more shortcuts

A default icon which is used everywhere that a site doesn't have an icon

Even faster startup!

Much more

Missing features
Rather then asking my friends what features they need the browser to have I ask them to try using it and then let me know the little things that drive them crazy. For me I discovered space key to page down and ctrl-up to move to the top of the page were killer features. Years of using those shortcuts were so ingrained that not having them was driving me nuts. I can surive without flash, but not those. So if you give the demo a whirl feel free to let me know what feature you use in your normal browser that you found missing.

Having an application that I use nearly everyday has been a lot of fun. QtWebKit is missing several things before this could turn into a browser I could use full time, most notably netscape plugins and disk cache, but already it has surpassed my expectations and brought joy to my weekend hacking.

Exporting History from Konqueror

2007-12-27T15:59:00.003-05:00

In my Konqueror settings I have had it retain my history sense August 2006. This has resulted in a history file that contains over eighteen thousand entries. As you can guess Konqueror has some trouble with a history of this size. A dataset of this size is a good way to test any history managers memory usage and performance. I was hoping to be able to export the data with dcop, but unfortunately the only function available is allUrls which doesn't make that much sense because the dates are just as important as the urls. To see your history urls with dcop use

dcop konqueror-11489 KonqHistoryManager allURLs

replacing 11489 with your Konqueror number of course. Reaching a dead end with dcop I whipped up a tool that uses libkonq to extract the history.


#include <kapplication.h>
#include <kcmdlineargs.h>
#include <konq_historymgr.h>

int main(int argc, char **argv)
{
    KCmdLineArgs::init(argc, argv, "konqueror", "konqueror", "I do stuff", "0.1");
    KApplication application(argc, argv);
    KonqHistoryManager manager(0, "manager");
    KonqHistoryList list = manager.entries();
    for (uint i = 0; i < list.count(); ++i) {
        printf("%s\n%s\n%s\n",
                list.at(i)->url.url().latin1(),
                list.at(i)->title.latin1(),
                list.at(i)->lastVisited.toString().latin1());
    }
    return 0;
}

Already this data has proved useful and helped me reduce my memory footprint by half and catch my performance bottlenecks so it can handle 20 entries or 20,000 just as fast. If you are interested in playing with the dataset send me an e-mail.

Valgrind callgrind tools Part 3: Code coverage

2007-12-22T14:41:00.003-05:00

In the Valgrind manual they list some tools you could write for Valgrind. One of the suggestions is to write a coverage tool. So a few months ago I sat down and did exactly that. It was a nice little tool and I happily posted it to the Valgrind developer mailing list only to discover that I was not the first person to do this nor was any of our solutions that good in the end. The easy part of the project is obtaining a recored of what was executed. In fact Cachegrind does a fantastic job of that. The only thing my tool did (that I thought was new) was automatically collect data on all the jumps which discovered after I wrote my tool was already present in the callgrind tool if you pass in the argument "--collect-jumps=yes". So me and several others have all written bad versions of callgrind and the cg_annotate script. Code coverage with Valgrind is made up of two problems. The first is obtaining the code execution and the second is applying this over the source for the program. The first is easy by using "callgrind --collect-jump=yes". The second part isn't as easy a new problem for every language, but at least for C++ using Roberto's parser makes this task a lot easier.

Here is an older output for a test that I wrote for QColumnView that shows some missing coverage:

valgrind --tool=callgrind --collect-jumps=yes ./tst_qcolumnview

./callgrind_coverage ~/dev/qt/tests/auto/qcolumnview/callgrind.out.16089 ~/dev/qt/src/gui/itemviews/qcolumnview.cpp


Function coverage - Has each function in the program been executed?
        Not executed: 612 void QColumnViewPrivate::_q_clicked(const QModelIndex &index)
        Coverage: 39/40 97%

Condition coverage - Has each evaluation point (such as a true/false decision) been executed?
443    for (int i = 0; i < ranges; ++i) {
842    for (; (i < list.count() && i < d->columns.count()); ++i) {
212    if (d->columns.isEmpty() || dx == 0)
976    if (!model() || !selectionModel())
976    if (!model() || !selectionModel())
229    if (!index.isValid() || d->columns.isEmpty())
229    if (!index.isValid() || d->columns.isEmpty())
272    if (leftEdge > -horizontalOffset()
336    switch (cursorAction) {
338        if (current.parent().isValid() && current.parent() != rootIndex())
338        if (current.parent().isValid() && current.parent() != rootIndex())
557        for (int i = columns.size() - 1; i >= 0; --i) {
549    while (currentColumn == -1 && parentIndex.isValid()) {
572    if (currentColumn == -1 && parent.isValid())
577    if (build && columns.size() > currentColumn + 1) {
580                             && !columns.at(currentColumn + 1)->rootIndex().isValid());
689    if (!columns.isEmpty() && columns.last()->isHidden())
689    if (!columns.isEmpty() && columns.last()->isHidden())
695    if (show && view->isHidden())
394    if (horizontalLength < viewportSize.width() && q->horizontalScrollBar()->value() == 0) {
813        if (!columns.isEmpty() && columns.last() == previewColumn)
1024    if (!model || columns.isEmpty())
1035            if (x != view->x() || viewportHeight != view->height())
1042            if (x != view->x() || viewportHeight != view->height())
368        if (diff < 0 && horizontalScrollBar()->isVisible()
877    if (currentParent == previous.parent()
877    if (currentParent == previous.parent()
879        for (int i = 0; i < d->columns.size(); ++i) {
893        for (int i = 0; i < d->columns.size(); ++i) {
512        if (!found && columns.at(i)->cornerWidget() == grip) {
        Coverage: 566/598 377%

Statement coverage - Has each line of the source code been executed?
        Coverage: 475/523 90%
-------------
Key:
 c == compiled
 e == executed
 r == return
 j == jump
 s == skipped
 - == comment/whitespace/not compiled
-------------
94      [j]  f: }
273     [c]:         && rightEdge <= ( -horizontalOffset() + viewport()->size().width())) {
286     [c]:             newScrollbarValue = rightEdge + horizontalOffset();
336     [j] f:     switch (cursorAction) {
369     [c]:             && horizontalScrollBar()->value() == horizontalScrollBar()->maximum()) {
579     [c]:         bool viewingChild = (!model->hasChildren(parent)
612     [c]: void QColumnViewPrivate::_q_clicked(const QModelIndex &index)
614     [c]:     Q_Q(QColumnView);
615     [c]:     QModelIndex parent = index.parent();
616     [c]:     QAbstractItemView *columnClicked = 0;
617     [c]:     for (int column = 0; column < columns.count(); ++column) {
618     [c]:         if (columns.at(column)->rootIndex() == parent) {
619     [c]:             columnClicked = columns[column];
623     [c]:     if (q->selectionModel() && columnClicked) {
624     [c]:         QItemSelectionModel::SelectionFlags flags = QItemSelectionModel::Current;
625     [c]:         if (columnClicked->selectionModel()->isSelected(index))
626     [c]:             flags |= QItemSelectionModel::Select;
627     [c]:         q->selectionModel()->setCurrentIndex(index, flags);
690     [c]:         columns.last()->setVisible(true);
726     [c]:         model()->fetchMore(index);
824     [c]:     previewColumn->setMinimumWidth(qMax(previewColumn->verticalScrollBar()->width(),
861     [rc]:     return list;
878     [c]:             && model()->hasChildren(current) && model()->hasChildren(previous)) {
939     [c]:             QItemSelectionModel *replacementSelectionModel =
941     [c]:             replacementSelectionModel->setCurrentIndex(
943     [c]:             replacementSelectionModel->select(
993     [c]:     QModelIndex br = model()->index(model()->rowCount(parent) - 1,
994     [c]:                                     model()->columnCount(parent) - 1,
1081    [c]:         opt.rect = QRect(option.rect.x() + option.rect.width() - width, option.rect.y(),

Note that this tool is a bit of a hack and is a work in progress so you will no doubt find issues. But it is good enough to be useful so I figured it would be time to publish:

The source for callgrind_coverage can be found in my callgrind tools git repository.

Valgrind callgrind tools Part 2: accessing information inside a callgrind files

2007-12-10T15:54:00.002-05:00

From Valgrind's callgrind manual:

Callgrind is a Valgrind tool for profiling programs. The collected data consists of the number of instructions executed on a run, their relationship to source lines, and call relationship among functions together with call counts. Optionally, a cache simulator (similar to cachegrind) can produce further information about the memory access behavior of the application.

The result is that the files that are produces contain a plethora of data just waiting to be mined. I have written a tool called callgrind_info to extract information out of a callgrind file.

When you want to watch a function for performance regressions callgrind excels at getting results that are similar one run to the next. Extracting the cost of a function in a nightly script would be a very useful thing to do to automatically determine if you are having performance regressions:

./callgrind_info callgrind.out.5511 -cost 'tst_QListView::cursorMove()'
158034263

The matching functionality for the "-cost" is the "-function" which will output every function that was used in some specific file:

./callgrind_info callgrind.out.5511 -functions tst_qlistview.cpp                              
tst_QListView::tst_QListView()
tst_QListView::~tst_QListView()
tst_QListView::cleanup()
tst_QListView::cursorMove()
...

callgrind_info also lets you extract a list of the different specification types.

If you want to know every object/library that was loaded during the run of the program:

./callgrind_info callgrind.out.5511 -spec ob
/lib/tls/i686/cmov/libm-2.6.1.so
/usr/lib/libstdc++.so.6.0.9
/home/ben/dev/qt/tests/auto/qlistview/tst_qlistview
/lib/tls/i686/cmov/libnss_nis-2.6.1.so
/lib/tls/i686/cmov/libc-2.6.1.so
...

Every file:

./callgrind_info callgrind.out.5511 -spec fl | more
???
qbasicatomic.h
qdatetime.h
qwidget.h
qlist.h
qabstractitemmodel.h
qsize.h
...

Every function:

./callgrind_info callgrind.out.5511 -spec fn | more
0x00003460
floor
QBasicAtomicInt::operator int() const
QBasicAtomicInt::operator!=(int) const
QBasicAtomicInt::operator!() const
QBasicAtomicInt::operator==(int) const
QBasicAtomicInt::operator=(int)
QTime::QTime()
QWidget::show()
QWidget::resize(int, int)
QWidget::height() const
...

The callgrind info tool just touches on some of the data that can be extracted, but already it has proved to be very useful.

The source for callgrind_info can be found in my callgrind tools git repository.

Valgrind callgrind tools Part 1: decompressing callgrind files

2007-12-09T14:37:00.002-05:00

If you have ever used the Vallgrind tool callgrind you will find that it generates a file "callgrind.out.pid". Looking inside of this file you will find a compressed version of the execution of the program. Here is a snippet from a callgrind file I generated last night:

fn=(43726) tst_QListView::indexAt()
463 4
+2 4
cfn=(24288)
calls=1 148 
* 3399
+1 1
+1 1
+2 4
cob=(26)
cfi=(265)
cfn=(21860)
calls=1 150 
* 320655
+1 5
...

After reading the callgrind format specification you shouldn't have too much trouble being able to read it if you really want to. By default the callgrind files are compressed which make it harder to read then it has to be. So if you get a compressed callgrind file that you want to read I have written a tool to decompress the file into a more reasonable format. Here is the same code as before, but this time decompressed:

fn=tst_QListView::indexAt()
463 4
465 4
cfn=QtTestModel::QtTestModel(QObject*)
calls=1 148 
465 3399
466 1
467 1
469 4
cob=/home/ben/dev/qt/lib/libQtGui.so.4.4.0
cfi=qlistview.cpp
cfn=QListView::QListView(QWidget*)
calls=1 150 
469 320655
470 5
...

If you are going to be either extracting information out of a callgrind file or writing your own tool that generates a callgrind files like my QScript profiler having a decompresser comes in handy.

The source for callgrind decompress can be found in my callgrind tools git repository.

Limiting the runtime of a program

2007-11-28T10:47:00.001-05:00

I have some programs that run in a cron job at night and occasionally they hang and don't exit which means that anything scheduled after them doesn't run and the next day I have processes hanging around. To solve that I made a twelve line program that uses features in QProcess to automatically kill a process that is taking longer then five minutes. Here is the slightly bigger version that lets you specify the timeout if you want and has a help.

timelimit -l 60000 ./reallyLongrunningapp

#include QtCore
int main(int argc, char **argv)
{
    if (argc < 2) {
        QTextStream out(stderr);
        out << "Usage: timelimit -l 1000 command [args]" << endl;
        return -1;
    }

    QProcess process;
    QStringList arguments;
    for (int i = 2; i < argc; ++i)
        arguments.append(argv[i]);
    QString command = argv[1];
    int timeLimit = 1000 * 60 * 5;
    if (argc > 3 && command == "-l") {
        bool ok;
        int userLimit = arguments.value(0).toInt(&ok);
        if (ok) {
            timeLimit = userLimit;
            command = argv[3];
            arguments.pop_front();
            arguments.pop_front();
        }
    }
    process.setProcessChannelMode(QProcess::ForwardedChannels);
    process.start(command, arguments);
    process.waitForFinished(timelimit);
    return process.exitCode();
}

QtScript profiling

2007-11-09T04:31:00.003-05:00

Working on some QtScript code a little bit ago I wanted to profile the code. There didn't exists a profiler to I wrote one. It uses QScriptEngineAgent which is part of Qt 4.4.

qscriptprofiler runs with script files as arguments that are each executed in the order they are passed. When the script is finished it will generate a callgrind file that can be loaded in KCachegrind. At this time qscriptprofiler only uses gettimeofday() to determine how long something takes so there is the possibility that it isn't accurate if your system is under heavy load. Improving it to not use the system clock is left as an exercise to the user (it is good enough for what I have been using it for). The profiler can be used in other projects and a pri file is included.

You can get the source for qscriptprofiler on github.

Auto test stub generator

2007-11-08T12:24:00.004-05:00

When writing auto tests for Qt code one of the more annoying parts of the task is getting the initial test file up and running. The basic makeup of the test files are about the same so I wrote a little tool that helps by automatically creating stub code for the class you are writing a test for. Then it is just a task of going through the file filling it in with the actual tests.

Some features include:
- Creates a stub tests for each non private function in the class.
- Creates a subclass for the class to expose any protected functions for testing.
- Creates a basic sanity test that just calls each function.
- Creates a _data() function for each test.
- Populates the _data() functions with columns for each argument and the return value.
- Adds QFETCH stub code that matches the generated _data function as a place to start from.
- Adds the four init and cleanup functions with documentation so you don't have to look up what does what.
- Adds signal spys to each tests if the class contains any signals.

The source code can be found: QAutoTestGenerator

Here is an example class:


class Example : QObject
{
    Q_OBJECT
signals:
    void done();
public:
    void launch(int value);
protected:
    bool crashed();
};

And the generated stub:


#include <QtTest/QtTest>
#include <example.h>

class tst_Example : public QObject
{
    Q_OBJECT

public slots:
    void initTestCase();
    void cleanupTestCase();
    void init();
    void cleanup();

private slots:
    void example_data();
    void example();
    void launch_data();
    void launch();
    void crashed_data();
    void crashed();
    void done_data();
    void done();
};

// Subclass that exposes the protected functions.
class SubExample : public Example
{
public:
    bool call_crashed()
        { return SubExample::crashed(); }

    void call_done()
        { return SubExample::done(); }
};

// This will be called before the first test function is executed.
// It is only called once.
void tst_Example::initTestCase()
{
}

// This will be called after the last test function is executed.
// It is only called once.
void tst_Example::cleanupTestCase()
{
}

// This will be called before each test function is executed.
void tst_Example::init()
{
}

// This will be called after every test function.
void tst_Example::cleanup()
{
}

void tst_Example::example_data()
{
}

void tst_Example::example()
{
    SubExample example;
    /*
    example.isDone();
    example.launch();
    example.call_crashed();
    example.call_done();
    */
    QVERIFY(false); // remove after test is implemented
}

void tst_Example::launch_data()
{
    QTest::addColumn<int>("value");
    QTest::newRow("null") << 0;
}

// public void launch(int value)
void tst_Example::launch()
{
    /*
    QFETCH(int, value);

    SubExample example;

    QSignalSpy spy0(&example, SIGNAL(done()));
    QCOMPARE(spy0.count(), 0);
    example.launch();
    */
    QVERIFY(false); // remove after test is implemented
}

void tst_Example::crashed_data()
{
    /*
    QTest::addColumn<bool>("crashed");
    QTest::newRow("null") << 0;
    */
}

// protected bool crashed()
void tst_Example::crashed()
{
    /*
    QFETCH(bool, crashed);

    SubExample example;

    QSignalSpy spy0(&example, SIGNAL(done()));
    QCOMPARE(spy0.count(), 0);
    example.call_crashed();
    */
    QVERIFY(false); // remove after test is implemented
}

void tst_Example::done_data()
{
    QTest::addColumn<int>("foo");
    QTest::newRow("null") << 0;
}

// protected void done()
void tst_Example::done()
{
    /*
    QFETCH(int, foo);

    SubExample example;

    QSignalSpy spy0(&example, SIGNAL(done()));
    QCOMPARE(spy0.count(), 0);
    example.call_done();
    */
    QVERIFY(false); // remove after test is implemented
}

QTEST_MAIN(tst_Example)
#include "tst_example.moc"

Quickies

2007-10-29T15:31:00.003-04:00

I have a number of presents for you all.

First off is a little file manager that I wrote several months ago. It is called Ambit and is pretty much a weekend clone of OS X's Finder.
http://github.com/icefox/ambit/tree/master

Robero's wrote a kick ass C++ parser and code model. With it you can quickly and easily parse C++ and walk it sources to be able to do all sorts of cool things. Older versions of Roberto's parser can be found in various places in KDE, but they don't compile with just QtCore (the one in KDevelop require kate!) so I pulled this one out of the Qt Jambi sources and bounbled it up, added an example and put it up online where others can easily get at it to make their own tools such as those I talked about in making testing enjoyable with parsers.
http://github.com/icefox/rpp/tree/master

I have begun learning Lisp (or more exactly Common Lisp). Rather then just trying to make something and read tutorials as I need them I am slowly going through some Lisp books I own, working through the problems and exercises. I started a blog with the notes as I learn Lisp. http://takentheredpill.blogspot.com/

Why working for Trolltech kicks ass

2007-10-17T16:41:00.005-04:00

An office with a door and window!
Do you find open air office cubes help you get in the zone for programming? Neither do I. Many people have written why offices are more productive then cubes so it is surprising that places like Google still use cubes. What is better then an office with a door? How about a window the size of the wall with a nice view. Not a view of another building twenty feet away, but hills, trees, and the occasional bird sitting on the ledge. At Trolltech there are two developers in every office (photo was taken during a move when there was only one desk in it).
Smart developers
Walking down the hall you will find many smart developers. They aren't just smart, they like what they do and many work on open source projects such as KDE and Webkit. How about putting Qt on the dreambox, hosting KDE hackathons on Sunday's, contributing to Git and much more. We love what we do and it shows.
Release source code
Go work for Google and there is a good chance you will never get to release your source code and perhaps never even be allowed to talk about it! Qt is released under the GPL license. Qtopia is released under the GPL. QScript, Qt-jambi, etc all GPL. We put up nightly snapshots for both our stable branch and main branch and host a matching git repository with them. We encourage releasing the source to our projects under a open source license. We put the source for our projects, neat ideas, and components up on labs.trolltech.com before they go in Qt so you can check them out. As a developer when you hack on code in Qt it doesn't end up in an internal black hole or locked in a binary, but is available for you to look at years later and reminisce or show off*.

*Of course that also means that everyone else reads your code too so customers that catch your bugs will often send in useful patches so remember to write full test coverage.
Company trips
In the fall the developers head off for a cabin trip. Those in the Berlin office drive up and bring some excellent German beer and food. We rent some cabins for a weekend and have a lot of fun. In the spring the entire company heads off for a fantastic ski trip. Trips often include many other events such as bike rides, hikes, snow man contests and on one occasion a sleigh ride.
labs.trolltech.com is a place where we can blog about what we are working on, put up code, projects and ideas. We get input on API and features and users get to mess around with what is coming down the pipeline and get solid information about where Qt and Trolltech is headed. As a developer you can talk about what you are working on, you can show of API and get feedback to make your product better.
Creative Fridays
You have probably heard of Googles 20% rule. At Trolltech we have creative Friday where you get to work on a creative project. Many of these end up labs.trolltech.com so even if they don't end up succeeding or not put in Qt they will still be on labs. Got an idea or itch to make something better that is outside of your day to day project then this is a perfect fit and many excellent improvements and features in Qt have come about because of this.

What I learned from Qt that made me a better programmer in C++

2007-10-08T00:50:00.001-04:00

Reginald Braithwaite says he would love to hear stories about how programmers learned from X that makes me a better programmer in Y. So here are a few quick thoughts on how the Qt library made me a better C++ programmer.

1) The API matters, a lot

In a way Trolltech is in the business of selling good API's. By making their API consistent across Qt, it is easier as developers to understand how the class works and one can often guess correctly what an API will be without having to look it up. You wont find functions that take six bool arguments or function names that are overly vague when a better one will do. Each new function gets multiple API reviews and before every release all new functions and classes get a final review. Check out this excellent article on designing C++ APIs for some good examples.

Every time you add a new class or function you are introducing new API. Even if that API is only for yourself it is something that you will have to use and later read. On personal projects taking two minutes to step back and cleanup the API will payoff in the long run. Your code will be more consistent and when someone else comes along to check it out they will not only be happier with the cleaner API, but have an easier time getting up to speed and more likely to contribute back.

2) How to hide your private data (or working around one of c++'s flaws)

When building libaries in C++ there are a lot of rules you have to follow to make sure that you do not break binary compatibility.

The first and best step you can take is to make sure that your API is good (see #1). This goes a long way in fixing design issues before you can't anymore. Walk though use cases and make sure that the class provides what is needed. Hopefully most of the time you should be able to get away with just adding a new function or two, but the core of the class will be stable.

But behind the public API there are tricks you can used to hide all of the internals of the class in a way that lets you change them at will without breaking binary compatibility. One of them is the private class:

class FooPrivate;
class Foo {
public:
  Foo();
private:
  FooPrivate *d;
};

A technique that is used in Qt and KDE each class has a private class (with its own header). Data, implementation and private functions are moved into the private object. This provides a number of very positive benefits.
- You can add internal virtual functions, change function names, and generally do whatever you want.
- The implemenation can be completly changed without worrying about breaking binary compatibility.
- Users of the class don't need to include all the headers for private objects, reducing compile time.

Seven Tips for giving Google Edu presentations (or any tech video that is going up online)

2007-08-18T20:31:00.001-04:00

I have been subscribed to the Google Edu Video RSS for about a year now. I love how one day there is a presentation about how to best defend against a cougar attack and the next day a presentation about LLVM 2.0 It reminds me of when I was at CSH and every week there was a presentation or two. On Wednesday "Compression algorithms" and Friday "How to make a potato gun". The fact that anyone could give (or request) a presentation at any time to a small audience led to a wide range of interesting topics. Maybe you didn't know anything about the topic, but you always learned something.

After watching many Google Edu videos I have put together things to do when giving your presentation.

Don't thank Google!
The video tape is only 60 minutes long and wasting two minutes of it saying thank you is a waste of time. Yes you use Google and yes you find it useful, we all do. You don't need to say it.

Don't give your life history
If it takes eight minutes to get to the start of your real presentation then you have done something wrong. Usually there is someone who introduces you in thirty seconds or less. You do not need to give us your resume. If you are on Google edu we will give you the benifit of the doubt that you probably know what you are talking about.

Repeat questions
Any time someone asks a question you need to repeat it. If you do not repeat the question then that whole part of the presentation is dead to those who are watching it online. For some presentations that are largely QA it means that you will only ever reach the ten people in the room and not the other ten thousand that are trying to watch it online.

A basic Intro is more then helpful, it is required
When giving a technical talk, spend one minutes explaining what it is that you are doing. Example are good too. Jumping straight into the meat without anything will loose a lot of people who will just click to the next video. We will probably be able to understand a lot of what you are talking about if you just give us some place to start from.

Don't poll the audience
The physical audience size for some of these presentations ranges from 10 to 50. You should not ask for a raise of hands about anything, because there are 50,000 other people who will view it online.

Don't give out secrets
Just because only eight people turned up to see the presentation doesn't mean that you can say "just between you and me..." and then give away your company plans [cough]Adobe dude[cough]. Remember that the physical audience is nothing compared to the number of people that will watch it online.

Don't feel bad because only 20 people show up
At the start of the Surviving in the wilderness video that is linked above the guys is disappointed because only twenty people show up, but little did he know that the video would be watched twenty thousand times. How is that for an audience?

Javascript Genetic Algorithm

2007-08-06T22:58:00.000-04:00

The other evening I was looking to play around with QScript. I have done a handful of JavaScript and know that there are a number of very interesting features that the language possesses. I have also been looking to hack on a Genetic Algorithm project. Putting the two together I made a Genetic Algorithm example written in JavaScript. It was quite a lot of fun to make and is quite a small amount of code in the end. I included an example that tries to find the shortest path between a set of points (the travelling salesman problem). You are free to type in any problem you want and it will attempt to solve it with the genetic algorithm. After it runs it will generate a graph showing the progress as its gets a better and better solution (using the canvas tag, sorry it doesn't work in Konq 3.x). And it will paint the final solution that it finds to the problem. Although it ended up being a little website rather then a Qt app it was fun learning more JavaScript and more about GA's. If you are interested in learning about GA's I have a number of links including some to Google books at the bottom.

Extend your MacBook battery life

2007-05-23T21:16:00.000-04:00

The PowerTOP project inspired me to take a quick look at what was running on my MacBook under OS X. Just by using the activity monitor (and top) I could very quickly spot some obvious issues.

- The world clock in dashboard (which is one of the four included and running ones by default) fires off once a second, I didn't really need that widget anyway.

- I had enabled access to disabled devices long ago when playing around, but it seems to start a process that wakes up a lot. Turning that off solves that.

- In OS X I often use iTerm, but it also is waking up all of the time. For the short term I will just use the built in Terminal, but being open source a look into the code might make for fun some day.

- If you haven't already downloaded and installed a add blocking css file that will block a lot of ads out there and then add to the css file to block flash banner ads which will run continuously and just suck up CPU:


/* this hides the usual 468x60 Flash banner ads */
embed[type="application/x-shockwave-flash"][width="468"][height="60"] {
  display: none !important;
  visibility: hidden !important;
}
/* this hides the not so usual but very annoying 728x90 Flash banner ads */
embed[type="application/x-shockwave-flash"][width="728"][height="90"] {
  display: none !important;
  visibility: hidden !important;
}

My crude tests (turn everything on, view estimated battery life left after idling for a few minutes, turn everything off and do the same) noticed a nice jump in expected battery life.* Of course this can't approach anywhere the improvements that powerTOP could give in Linux, but it was surprising how much of a difference I was able to get with give minutes and top.

*Of course it is highly likely my testing method could just be a fluke, feel free to run your own test.

Hair Cut

2007-04-19T01:52:00.001-04:00

Yesterday I got my hair cut short for the first time since my wedding five years ago. I have been thinking of getting it cut off for about six months and now seemed a good time. I donated my hair to Locks Of Love which is an organization that provides wigs for kids. From their website: "Most of the children helped by Locks of Love have lost their hair due to a medical condition called alopecia areata, which has no known cause or cure." Since getting it cut I have caught myself trying to play with it when thinking about thing. I guess it will take a few days to forget that muscle memory. Of course a blog like this is pretty much worthless without the before and after photos:

Anigma Game

2007-03-18T03:41:00.002-04:00

Originally called "Puzz-le", Anigma was a very well done puzzle game for the Sharp Zaurus. When blocks of the same color are moved next to each other they disappear. The goal is to eliminate all the blocks on each level. Beside colored blocks there are other various objects the player can interact with. This includes elevators, disappearing trap-blocks, fire pits and more.

Released under the BSD license I ported it from Qt 2 to Qt 4. The original source is available on Walter Rawdanik's website. After porting the code some parts were upgraded to take advantage of new features in Qt4, but overall the code retains the original design. If you enjoy this game please send Walter Rawdanik an e-mail thanking him for licensing Puzz-le under the BSD.

The source for Anigma is under the BSD license. I recommend using Qt 4.2.3 with the game, but if you use 4.2.2 everything plays fine just the text at the top of the screen shows up very dark due to a Qt painting bug.

Download Anigma on the Anigma project page.

Git and hooks

2007-03-15T03:05:00.002-04:00

Update: See my more recent blog post Git hooks for a more in depth look at Git hooks.

The past few weeks I have started learning Git and this past week started using it for one of my own little project I have been hacking on. It has been a fun experience and once I figured out the basics it went pretty smooth. Here are a few tips that would have been nice to know before I started:

- Don't bother using Cogito, but just use Git directly. Git has matured a lot sense it was first introduced and removing another layer helps to make it less confusing.

- There are a handful of docs all over, but Git introduction/tutorial is the best intro I have seen.

- The equivalent of "svn revert file.cpp" is "git checkout HEAD file.cpp"

- Unlike svn Git wont automatically know what you want to commit, you have to add the file first.

- Git bisect is a handy little thing.

- repo.or.cz provides a nice easy place to host mirrors or push locations.

The one thing I haven't figured out how to do is go back and edit older commit messages which you can do in svn.

But the coolest thing about Git has to be the hook functionality. CVS, Subversion, Perforce, and well all revision control systems have hooks, but they are all located on the server side, there are no hooks for the client. With Git because your depot is on the client suddenly any hooks that you add are all run locally. To add a script to Git you place it in .git/hooks/ and modify the appropriate hook such as pre-commit to utilize it. Any time I have ever seen hooks being used before now was for integrating with other project services. KDE has some hooks so you can say that the commit solves a bug and it will automatically e-mail that change to bugs.kde.org. Subversion comes with a script to e-mail the change to someone or presumably a mailing-list. KDE has one that is more advanced, but still follows the basic idea of summarizing the change and letting someone/something know. From time to time in open source projects I have heard developers discuss the idea that the hook system can be used to make sure that no commits break the build by compiling the project with the patch first, but invariably it is never implemented because usually the build would take longer then the average time between commits, or the server is on a slow machine, or that the server for security reasons isn't setup for package building, or that the server doesn't have space, or sometimes devs want to commit broken code. Another common request is that any commit go through a code style tool. At least in KDE this would never fly because the KDE repository covers many project with different coding styles just for starters. Suffice it to say any hook that is put in place has to go through the community and overall very inflexible.

With the repository on my local machine I was quick to take advantage of that fact and added a handful of scripts to my pre-commit to:

- Check that the code in the patch conforms to the code style:
"astyle --indent=spaces=4 --brackets=linux --indent-labels --pad=oper --one-line=keep-statements --convert-tabs --indent-preprocessor "
- Check that the project compiles (In theory it was compiled before committing so this should be quick)
- Check that all the autotests affected by the patch pass

My development machine is a fast machine, no one else will be wanting to commit to the server, and I know my box is properly setup for building the repository. The above hooks usually are very quick and I think whatever delay it causes is worth it in the long run. The personalization of the hooks is what makes it more powerful then the hooks on a server though. For me if the source is already built I am not going to force it to distclean and rebuild while for someone else this might be required. Beyond that I even made a hook that checks what time it was and if it was after 3am then it tells me to go to bed and review the commit in the morning. A hook like that would never get into the KDE repository, but it is something I can have locally. Making quick modification to hooks are not something you would do with them on a CVS server. Hooks in revision control systems generally have been a very inflexible and an underutilized idea.

Dozens more ideas for hooks can be found on englishbreakfastnetwork.org. Spell check docbooks, validate xml files, check for binary incompatibility, etc. Because most of the time patches are small most tests wouldn't even do anything because they wouldn't be affected (i.e. if no xml file have changed there is nothing to validate).

Once I had it setup and running looking at Subversion is seems unfortunate that there isn't an official way to add client side hooks there too.

Some more links:
Hooks in Git Documentation
Hooks in Subversion Documentation

QColumnView

2007-02-28T05:48:00.001-05:00

With 4.3 features wrapping there is a new itemview class called QColumnView. It provides a way of displaying a tree model. Each column is another level in the tree with the far left being the top of the tree and the far right side being a node. It is known for quick and easy keyboard navigation and a final preview area. Most commonly this view is known for its usage in OS X and in particularly in Finder. Below is a screenshot of using the view with the dir model and a simple file preview widget.

This widget isn't just for the finder though. I have found the same ui pattern in a lot of places (mostly on OS X). On extreme example was a date scheduling application. The first column let you select the month, the second you selected the day and the third the time. But rather then using lists of strings it showed calendars and clocks. I have seen a lot of applications who only need a few columns such as iTunes with their genre/artist/album columns. There are other applications who have only had two columns, the model was just a list (which was the first column) and when you clicked on an item the second column showed a detailed control of the item.

From the UI Patters and Techniques page on Cascading Lists:

"Use When the user needs to navigate a hierarchy that isn't very deep, but might have many items on each level. An [QTreeView] would work, but the user would be scrolling up and down a lot to see all the items, and they wouldn't get a good overview of the items at higher levels in the hierarchy. ... By spreading the hierarchy out across several scrolled lists, you show more of it at once. It's that simple. Visibility is good when you're dealing with complex information structures. Also, laying the items out in lists organizes them nicely -- a user can more easily keep track of what level they're dealing with than they could with an outline format, since the hierarchy levels are in nice predictable fixed-position lists."

And of course QTimeLine is used to provide smooth animations when scrolling between two columns. I snagged an animated gif, but be warned it came out a lot choppier then in real life.

The most intersting part of QColumnView is the last column. An empty space by default it can be replaced with a large widget that provides controls or previews. With such a large area to use quite a lot of information about the currently selected item can be shown. Unlike a tree where a separate area is needed or with icon where there is a constant battle to keep the items small, but have then be big to convey the information (often being just a small thumbnail). I look forward to seeing how it is used.

Qt QAbstractItemModel ModelTest

2007-02-27T14:17:00.000-05:00

With labs.trolltech.com going live this week I was able to release a little tool that provides a way to check for common errors in implementations of QAbstractItemModel.

Never really finding anywhere in Qt where this would belong, labs is a perfect way to get this to developers. Although quite small it has been very usefull in finding issues, a lot faster then me sitting down and reading through code. It is especially usefull when writing tree based models because it will check all of the edge cases. On many occasions when KDE developers have asked me questions about problems they are having with their models, after running their code with the model test I was able to resolve the problem within minutes.

You can find the source on its labs page here:
http://labs.trolltech.com/page/Projects/Itemview/Modeltest

PixelTool

2007-02-24T17:22:00.001-05:00

Included with Qt 4.2 you will find a useful cross platform application you might not know about, called pixel tool. Pixeltool is a desktop magnifier and as you move your mouse around the screen it will show the magnified contents in its window. Hitting the space bar freezes the application so it will stop updating. The application source is located in /tools/pixeltool/, and is built by default so the binary is placed in bin/ like assistant and designer. Below is a screenshot selecting the plus in the tree menu in KMail.

If you right click on the window there are a handful of features you can tweak such as the grid color, grid size and zooming. I am not too sure if any distributions package this tool (Debian's qt4-dev-tools package doesn't), but it would be cool if they did.

Home For Christmas

2007-01-15T03:05:00.000-05:00

For christmas I flew back to Boston to be with relatives. Having lived in America my entire life up until recently it was an interesting experience and I was surprised by the differences in American culture that I noticed upon returning.

After not owning a car the last year and a half I was amusing at how often people would drive when they could walk. My favorite experience the past few weeks was when I went out to lunch with some friends. We pulled out into the road, which was the main road for the town and the traffic on the road was almost bumper to bumper, stop light to stop light. Five minutes later about a mile up the road we pulled into the restaurant. It was like in Office space where the guy with the cane is walking faster then the cars. The best part was that the car I rode in was a hybrid and during the middle of lunch we briefly talked about the price of gas at the pump.

Jen and I walked around town, but still it was surprising how often we had to convince people that we really could walk a mile to the store and didn't need a ride. Someone once told me that the streets of Europe are not as handicap accessible as in America, but at least in Arlington that wasn't true at all. Lots of sidewalks don't go down to the road on corners, trees have uprooted chunks all over the place, many side streets simply didn't have sidewalks, and some side roads had no sidewalks and barriers on both side (so you couldn't even walk in the dirt) highly discouraging anyone from walking that way. One day it was sixty degrees or so outside and we were happy to see other adults outside until we realized that every single one was in some sort of workout clothes and running or "quick" walking. As far as I can tell walking in America is only for exercising, not for actually getting somewhere.

Although I knew that there was less adds on TV back home in Norway I was surprised just how many there actually where. Every few minutes, another set started for your enjoyment. I watched the evening news a few times and was annoyed at just how poor the news is here. The things that they report that are not important or the lack of actual news in a five minute span was amazing.

Part of the fun of visiting was eating all of the food I missed, I wonder if Norwegians who live in the America miss brown cheese? Every other day we went to a different old favorite restaurant with friends. The food was fine, but I think in my head I had been looking forward to it for so long that they couldn't match up to my expectations. One in particular I was looking forward to was Mountain Dew. I it different tasting over in Europe, but of course by the time I came back here it doesn't taste as good as I remember.

What would going to the US be without visiting a mall and the local megastores. Apple stores sure seem to be doing well and I couldn't help but noticing every time I was in one there were more girls then guys in them. As some of you know I enjoy transformers so it was fun going to the toy stores where after such a long time they were all new to me and I couldn't help but pick up one or two. Similar to the malls we went to the local grocery store. The amount of variety and over abundance of "stuff" at places like super stop in shop is amusing. Having become accustom to shopping in stores who's total size was only the size of the produce section at stop and shop I really had to question if getting to choose between two hundred tooth brushes had any benefits.

During this whole time the closest I got to programming was getting a new MacBook. I have to say I am very happy with it. The speed jump from my old 667Mhz powerbook is amazing and I can actually do real work on it (rather then sshing somewhere else). I like the built in iSight and set it up to take a picture every time it wakes up from sleep after a set amount of time which has presented me with a range of amusing photos, such as the one on the left. Touching on some of the software I must give credit where credit is due, iMovie/iDVD is slick, really slick. I video taped Christmas and with just two hours work pulled it off the camera, did basic editing, titles, transitions, and slapped it on a DVD with photos also taken during the video taping.

One task I got done while here was closing my savings account with Bank Of America and opening one with ING direct. Besides having a horrible website, Bank of America's savings accounts have a dirt floor low interest rate at .5% last I checked verses ING's 4.5%. If you are interested in opening an ING direct account send me and e-mail and I can send you a referral e-mail which will give you $25 just for opening the account. A nice easy way to get a better return on your savings and a little bonus.

My Dream File Manager

2006-12-01T03:00:00.000-05:00

What follows is a collection of points for my dream file manager. It was collected from notes I have been taking over the past two years so I am sorry if it doesn't flow that well together. So grab a beer and get ready for some chuckles as you read the key points for what would makes my dream file manager.

1) Hide the details from a user on how to access their files

Files are stored on volumes. Users should be able to just point and click and have access to these volumes. "Volumes" is a generic term referring to a number of things such as cameras, ipods, nfs shares, ftp, fish, zip, and more. Hardware volumes that are removable should just appear in the file manager with no effort on the part of the user by default. Software volumes must be discoverable (through a network explorer) and if possible accessible through an assistant such as in mockup on the right.

Currently in Konqueror you need to be taught about the useful ioslaves by someone else before you discover them yourself. This would not have been a problem if there was the most basic assistant to help users discover what ioslaves were available.

To accomplish this first task the file manager should have tight integration with solid and HAL to get all new hardware volumes and possibly provide an assistant to make accessing software volumes discoverable. For network browsing the mdnsResponder is also available for discovering local shares.

2) View & Edit meta information

Files contain meta information that is for the most part only relevant for a file manager. This can be broken down into two categories, the first being for volumes and the second being for files.

Volumes:

Volume Name
Free space
Size
Health status
etc

Files:

Size
Permissions
Ownership
What opens this file?
Icon
etc

The above two lists are very small and will change depending upon the file system, hardware, OS, etc. A modular approach is best.

Beyond the basic file system information file manager often have a plugin system for extended attributes editing such as the comment, title, and keywords on a JPEG or the rating of a mp3. This should not be part of the core file manager, but through an addon script system. The extended attributes should not provide means to edit metadata that would not normally be accessible to a type specific manager. For example in iTunes they don't let you edit the play count even though this is just another field in the id3 metadata. Extended attributes also by default shouldn't let you edit every possible metadata (image exif has quite a lot for example), but only the most common and useful.

3) Export and Import

A file manager can be expected to move files in and out of the current environment. Two common cases includes audio cd import and data DVD burning. This builds upon #1, but typically has a intermediate step that needs to be performed before a file can actually be copied/moved. In #1 you could see a CD and could copy the CDA files off the disk, but now you could see mp3's and when you copy them it will transparently encode the CDA files. Also creating a virtual folder that would then be burned to a CD or DVD. The export and import will typically be plugins that are well integrated with the interface (and not klugy addon hacks). Check out the Finder's Burn folders for an example. Another example is an iPod who when mounted is just a bunch of garbled files, but with a import layer the user could simply see a list of files with the correct file names and could drop new files in and copy old files off. Finally Trash is a nice little virtual folder that has the extra feature of emptying or restoring.

To accomplish the above today you can use dbus with k3b for the ripping and burning.

4) Basic information, but not a full blown viewer

The simplest example of this is showing a image thumbnail rather then the png icon. Other examples of this include showing a specific application icon rather then a generic binary icon. Depending upon the interface if there is a preview zone it can include extra basic information such a image dimensions and for audio basic playback.

5) Full search capability

File managers much have full search abilities, but by default only present a single line edit in the upper right of the interface. Through an assistant the user should be able to make virtual folders that contain rich search results. Some common useful virtual folders should be present on the system such as recently launched applications or files created in the last week.

Today Konqueror can integrate with strigi for search results, but Konqueror can even present the slocate ioslave until strigi is up and working.

6) Manual Grouping

To the user manual grouping is very similar to search folders. They are virtual folders that users can drop files into. The files are not moved there, but only sym linked. I hate to mention it because of the one and only one way that browsers have implemented it, but a common way that this is seen today in application is a bookmark system. To help you to see it differently think about how in iTunes you can have playlists that you build up yourself by dragging files from your library. You can have smart playlists (all songs I rank 5) and plain old playlists which you manually manage (the Norwegian audio lessons I am currently working on). A even simpler implementation of this is the OS X Finder sidebar which lets you drop folders and files on the sidebar for quick access later.

7) External interface to access the "stuff" in the manager

The first aspect of this is to be able to communicate with the file manager to perform operations such as a search, viewing recent folders, icons for files, etc. dbus is currently the solution for this. The second part is once you get the results being able to access the data which KIO/QFile do fine.

An basic example of this is a screensaver that queries the file manager for a list of the fifty most recently created images.

8) Extensible

A file manager should be extendable. Because of the wide use cases that this application is going to be used it needs to be modifiable. This can be as simple as providing a way for people to add right click actions to replacing the search system entirely.

Almost all of the above requirements for a file manager would benefit from plugin type systems and in fact many of them could be implemented in with them.

Incorporating the ability to write portions of the application in PyQt, kjs, QScript, or other scripting language should be thought about from day one.

Stressing this even more it is often time the community who when provided access to an extensible scriptable system will come up with features and capabilities that were not on your roadmap. A perfect example of this is FireFox that has countless addons. The features provided by the community will satisfy user itches without forcing the core application to become bloated while at the same time increasing the value of the application. Another benefit is that the interface can start out simple and only the most successful features can be added at a later date helping to prevent bloat.

9) Addon/intigrated tools/actions that build upon the above features

A file manager should provide by default a set of basic set of core features that enhance file management tasks.

History of everywhere that you have browsed, i.e. back/forward buttons.
compress folders & expand compressed files.
Wildcard / RegExp selecting.
.hidden file support that hides files/directories in the file system.

Some extra addon module/scripts can be provided for more advanced users

Drop Stack (drop your file(s) in this space, change directories and then drag it to its final destination)
console at the current location for advanced users.
renaming assistant.
Some sort of built in automatic backup system.
Ability to selectively mark folders for sharing.

10) Interface

Finally, after all that we get to the interface. The simpler the better, it should have tabs, but not on by default, there should be a search field, but only a single line edit in the top right. There Should be a single sidebar on the left that contains all mounted volumes (virtual or not), bookmarks, and virtual folders. Somewhere always present it should list the free space for volumes. As for the files there is merits to having only an icon view and a details view, but some would argue that a tree view and column view do very well. It would be fascinating to see several completely different approaches that focus on either a list, tree, or both. I will leave that up to a usability study, but no matter what the result it should be fully accessible from the keyboard. I personally love the column view in the finder for its quick keyboard access and final preview window. Also the first nine requirements generate a basic list of features that the interface should have, but they are not that interesting to debate.

Netflix Prize Contest

2006-10-29T12:39:00.001-05:00

I was very excited when the Netflix contest was first announced. I have been toying with that problem for a while, but I never had enough data to do anything really with so it was always a minor project of mine. As I watched people get interested in the contest many of them were spending the first week figuring out how to manage all the data that was given to them. The worst part was that after a week of doing this it wasn't very fun and they would stop, but yet they hadn't even worked on the real problem yet! I have been staying up late this week, but having a blast and created a little present for you all, the NetflixRecommenderFramework. It lets you very quickly start hacking on your algorithm ideas without having to bootstrap everything else first. It even includes autotests. Maybe you can win the million dollars.

On the Qt side of things one of the included tools is a small GUI application that can be used to explore the database. At over 100 million rows I am happy to say that Qt QTableView can handle it just fine and there is no startup delay. This is a followup to last years blog entry 66 million rows in QTreeView where I got it up to 66 million rows without any work. This time around I have actual data behind each row and column and not just a model that generates the data based upon the row and column. The bugs found before were still there if I set the height of the rows too large at which point their total height was greater then max int, but I have fixed those issues and they will be in the next Qt release. Anyone know if Gtk can do this?

Update: fixed link to project and source.

Dynamic Models

2006-10-10T19:31:00.001-04:00

When creating a custom tree model in Qt one common request is to lazily populate the tree. Starting in 4.2 there is a nice way to do this using QAbstractitemModel's fetchMore(). In your model overload hasChildren(), canFetchMore() and fetchMore(). hasChildren() should return true for those index's that have children even if the rowCount() is currently 0. The default implementation will return true if rowCount() is greater then 0. QTreeView uses hasChildren() and not rowCount() to determine if it should draw the little triangle/plus symbol to expand that index. When the user clicks to expand the index QTreeView will call "if (canFetchMore(parent)) fetchMore(parent);" Taking advantage of this inside the model's fetchMore() implementation, the node can now be populated.

I have created a small example FtpModel which wraps QFtp. It will only list the directories that you have opened. It doesn't implement renaming, drag/drop, or any convenience for uploading/downloading files, but those can be added without much work if you want to take this and use it. Feel free to download the source and check out the model code. It is released under the BSD license.

On the right is a screenshot of the example application with a dirmodel on the left showing my home dir and on the right the ftp model connected back into my local box and a open directory.

Drill Down View

2006-09-25T15:17:00.002-04:00

When dealing with data that is in a hierarchy, one way to present the data is inside of a drill down view. Some examples of what a drill down view are include the recent Start menu redesign put out by SuSe, the TiVo menus, and the iPod interface. It has the fantastic features of taking up almost no space and being very intuative. For more in depth discussion on this view check out One-Window Drilldown on the UI Patters website. Creating this in Qt 4 was very simple, here is a animated gif screengrab of the results.

The implimention was done by subclassing QListView and its moveCursor function. Before changing the root index I would store the current view image. When the current index changes I start the animation. One feature found in drill down views is animations. In Qt 4.2 there is a new class called QTimeLine. This class is perfect for helping you add animations to widgets such as the above one. Below you will find the code that you can use to have a drill down view in your application. The only code I didn't show is a item delegate that I wrote to draw the triangles on the right hand side, but i'll leave that part as an exercise for the reader.


#include "QtGui/QtGui"

class DrillDownView : public QListView {
    Q_OBJECT

public:
    DrillDownView(QWidget *parent = 0) : QListView(parent) {
        connect(&animation, SIGNAL(frameChanged(int)), this, SLOT(slide(int)));
        animation.setDuration(200);
    };
    QModelIndex moveCursor(CursorAction cursorAction, Qt::KeyboardModifiers modifiers);

public slots:
    void currentChanged( const QModelIndex ¤t, const QModelIndex &previous );
    void slide(int x);

protected:
    void paintEvent(QPaintEvent * event);

private:
    QTimeLine animation;
    QPixmap oldView;
    QPixmap newView;
    int lastPosition;
};

void DrillDownView::paintEvent(QPaintEvent *event) {
    if (animation.state() == QTimeLine::Running) {
        QPainter painter(viewport());
        if (animation.direction() == QTimeLine::Backward) {
            painter.drawPixmap(-animation.currentFrame(), 0, newView);
            painter.drawPixmap(-animation.currentFrame() + animation.endFrame(), 0, oldView);
        } else {
            painter.drawPixmap(-animation.currentFrame(), 0, oldView);
            painter.drawPixmap(-animation.currentFrame() + animation.endFrame(), 0, newView);
        }
    } else {
        QListView::paintEvent(event);
    }
}

void DrillDownView::slide(int x){
    viewport()->scroll(lastPosition - x, 0);
    lastPosition = x;
}

QModelIndex DrillDownView::moveCursor(CursorAction cursorAction, Qt::KeyboardModifiers modifiers)
{
    if (animation.state() == QTimeLine::Running)
        return QListView::moveCursor(cursorAction, modifiers);

    QModelIndex current = currentIndex();
    if (cursorAction == MoveLeft && current.parent().isValid()) {
        oldView = QPixmap::grabWidget(viewport());
        return current.parent();
    }

    if (cursorAction == MoveRight && model() && model()->hasChildren(current)) {
        oldView = QPixmap::grabWidget(viewport());
        return model()->index(0, 0, current);
    }

    return QListView::moveCursor(cursorAction, modifiers);
}

void DrillDownView::currentChanged(const QModelIndex ¤t, const QModelIndex &previous) {
    if ((current.isValid() && previous.isValid())
        && (current.parent() == previous || previous.parent() == current)) {
        setUpdatesEnabled(false);
        setRootIndex(currentIndex().parent());
        setCurrentIndex(currentIndex());
        executeDelayedItemsLayout();
        // Force the hiding/showing of scrollbars
        setVerticalScrollBarPolicy(verticalScrollBarPolicy());
        newView = QPixmap::grabWidget(viewport());
        setUpdatesEnabled(true);

        int length = qMax(oldView.width(), newView.width());
        lastPosition = (previous.parent() == current) ? length : 0;
        animation.setFrameRange(0, length);
        animation.stop();
        animation.setDirection(previous.parent() == current ? QTimeLine::Backward : QTimeLine::Forward);
        animation.start();
    } else {
        QListView::currentChanged(current, previous);
    }
}

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    DrillDownView view;
    view.setWindowTitle("Use arrow keys");
    QDirModel dirmodel;
    view.setModel(&dirmodel);
    view.show();
    return app.exec();
}

#include "main.moc"

Transformers

2006-08-14T17:00:00.000-04:00

After years of collecting I have finally put together a little Transformers generation one site, ToyBin.org. The site is a complete gallery of every toy, its modes and accessories in 1984, 1985, and 1986. This includes such old favorites as Optimus Prime, Megatron, Soundwave, and Devastator. It also has a simple search system which lets you do things like search for Green Truck and see all of the photos that are that. On more then one occation I have had people tell me the color of a transformer they remember having as a kid, but they remember anything more specific. There is a rss feed for the Transformer of the day which has been quite a lot of fun being surprised by old toys you forgot about such as Astrotrain or Shockwave.

Network

2006-05-27T03:00:00.000-04:00

Like a lot of people these days I own more then one computer. Back in the mid ninities along with many others I would list my computers down to which hard drive was in what box, but that type of information changes quite often and provides almost nothing of value to the reader. Instead I will highlight a few items that I feel are important and why.

KVM Switch

Even back in 1997 I knew I needed something similar to what this product does. Once I started to have more then one computer I knew that having three monitors, three keyboards, and three mice wouldn't work out. It would be too expensive and take up way too much room. After doing a little research I purchased an OmniCube which is made by Belkin and have used it ever since. It allows me to have one monitor, one keyboard and one mouse attached to up to four computers simultaneously. To switch between computers I hit "Scroll Lock" twice in a row and then the keys 1 through 4. I would recommend a kvm to anyone that is looking to own multiple computers while keeping costs and space down.

Kinesis Keyboard

I tried out many different and exotic keyboards, but in the end I settled on a Kinesis Classic keyboard. This keyboard which has the most frequent keys such as enter, space, delete etc in the middle of the keyboard reduce the amount of work on the weakest finger, the pinkie. Due to its curved design my wrists don't bend reducing any possibility of injury. I also found that after a week I was able to type faster with this keyboard.

Herman Miller Aeron Chair

I code up to eight hours a day, seven days a week, and spend another twenty hours a week working on other projects in front of my computer. If my butt is to be plastered for that long in something it should be comfortable and wont break. What can I say? This chair isn't the best just because of hype. When I first got the Aeron chair everyone around the office came and took a quick spin in it. Sitting in the chair for a minute they would all say: "Ok this is comfortable, but is it really worth that much?" One day After working all day I attended a meeting and noticed just how good my chair was and how bad all of the other chairs were. You don't know how good something is until you have to use something much worse.

Quiet Computing

As computers get faster and hotter the fans get louder. When upgrading my Tivo's hard drive it was recommended to get one of the ultra quiet Seagate hard drives. When I installed it I found to my suprise that even though it was faster then the old Tivo drive it was dramatically quieter, and the old drive was quiet by any standard. Impressed and hearing of other quiet computing projects I started looking them up online. One of the items I discovered was the Antec Sonata case. With hard drive noise dampners, a very large (thus spinning slowly) rear fan that was temperature controlled and a power supply fan that was self controlled it was the prefect case for reducing noise. When I needed a new hard drive I got another Seagate and also picked up the Antec Sonata case. When my other computer is on I can't hear anything from the Sanota computer over the noise from the old computer (before it was upgraded). With a little research a few dollars more you can pick up very quiet computer components. You do not need to have a jet engine under your desk.

The router appliance

One of the big reasons why I got into Linux back in 1997 was that I was able to take an extra computer and with Linux's built in capability turn the computer into a router. Then all my computers could be online at once. Back then the other option was to either use wingate (a primitive proxy) or some overly expensive windows program. I learned a lot about Linux just by setting up all the different servers such as iptables, dhcp, and samba. But these days rather then having to run a whole separate computer running all the time and needing to know a lot about the inside of Linux anyone can walk into a computer store and pick up a low power, noiseless box that frankly does more then I had ever set up manually, all for less then $100. And the best part? It still runs Linux.

Mini's

2006-04-17T12:07:00.001-04:00

Over on slashdot there is a review of the Nokia 770 Internet Tablet. Maybe Nokia has big plans, maybe they are going to re-use some of the software (cough only the browser cough) for their phones, who knows, but I feel like a time warp back five years with the Zaurus. You could almost read slashdot and swap the name of the product and it would be the same. People complaining about the software, but saying "it doesn't matter because the community will quickly make it better/fill X nitch/create whole new solutions/create new better rom/fix X problems. It is really sad too because the hardware itself isn't even that great from what people are saying. So in a year or two this device will be dead. Maybe a whole new better device will come out or more likely three more revisions will limp out, only time will tell.

In a similar vain I saw a review of PyQt and BlackAdder and thought I would check it out because I am interested in what they would have to say about BlackAdder. I have never used it myself, just curious about the product I have heard about from time to time. Suffice it to say the reviewer found it less then stellar, with this choice quote summing it up:

TheKompany is going to have to bend over backwards to show they've mended their ways before I would ever consider buying anything from them again.

Not to end on such a unhappy note I recently found a LISP and Qt4 integration attempt. It is pretty cool and I got it working on my box without too much trouble. Good for quickly hacking up some test apps.

Documentation on the web

2006-03-12T18:30:00.000-05:00

There is something very interesting about web applications that desktop applications lack. That is integrated help. When you go to amazons page for Pragmatic Programmer (good book btw) notice on the right hand side in the blue box there is a link to learn more about how A9.com users save 1.57%. Or if you go to a random product on OldNavy.com you will notice that there is a link for a size chart. Head over to Slashdot and make a comment or most every online blog and you will often find a list of html tags you are allowed to insert, hints, and tricks on how to make your blog entry 'rich'. Sites like pbwiki.com provide an easy way to learn the wiki syntax. And if you go to wikipedia you will find a whole toolbox of links to documentation about thing you might want to do with that particular page. And to use 30boxes again they provide right on the main page an example of how to add an event.

Because the interface for a page in a web application takes up the full size of a web browser and the web application is mostly text (rather then buttons, lists, and menus) it is easy to think about adding help into the page. Not only can you integrate help into the webpage, but you can have the help be determined upon where you just were and what your current task is. And to top it all off because the help is right there users will actually read it unlike most application documentation which is read by so few users. Part of the reason might be that with web applications you can review the logs and see that everyone stops using your site on page X. Adding a tiny bit of docs to that page then gets everyone past it. As a developer having the docs embedded within your application I bet you would find yourself caring about it a lot more too.

KDE actually has a built in mini-documentation features, called WhatsThis. Of course I think you all will agree that WhatsThis is used probably less then someone reading the actual documentation. As sad as it is tooltips provide the majority of documentation for users. Some widgets do provide more such as the OS X search widget actually has some text inside it telling you what it does. If you goto digg.com you will notice that same thing in their search box. Browsing around in applications I actually had a hard time finding much embedded documentation. The closest thing is toolbars that have text under them. Maybe clippy has something to do with this. There was such a backlash against clippy that no one has dared try anything similar. Today many believe that there is some magical interface that is so user friendly anyone can use it without any documentation no matter how complicated the task.

Here are just a few simply ideas on how documentation could be more integrated with the application:
- Now that we have opengl windows you could flip a window over and on the back would be the documentation for it.
- A drawer that would contains mini paragraphs about the current operation and links into the docs for more information.
- Clickable lables with links into the documentation or even just pulls up the documentation in a drawer.

Of course these are just some ideas off the top of my head. User interface testing would be a key aspect to finding out what would work best. Suffice it so say when you use online applications that has embedded documentation it makes the entire experience easier and better.

Lets have a meeting

2006-03-12T16:30:00.000-05:00

It is amusing to watch the new internet calendaring companies as they fumble around. Yahoo has one, Google now has something called C2 and there are many other sites out there such as 30boxes which all seem to do a good job as a calendar. Honestly though, most people don't really care to have a calendar. For those people that want to write out their schedule they will use a calendar on a piece of paper in their back pocket or a PalmV (which still rocks btw). You might convince a few of these people to be your customer, but after that you are dead in the water and have no revenue.

People keep creating calendaring applications because you typically use calendar apps to schedule to meet with other people in the office. That is where the tide of people are hiding (and money). The main reason you want your calendar online is so that others can find out when it is best to meet with you. Once you get one person using your site to schedule meetings then others might use it more and soon it will be a little virus sucking more and more users into using it because it provides a valuable service of having everyone else's calendar and a easy way to schedule. If you made the front page of your "calendar" site be four lines listing your e-mail/im, the person you want to meet with, date, reason and a big schedule button you probably would do better then a lot of calendar sites out there today. Google can no doubt leverage gmail and gtalk to make an easy scheduling system (and oh yah have a calendar too). If you go to 30boxes you can tell that scheduling a meeting is much more of just another feature and not the central part of the application.

What is the most surprising thing about meetings is that it is an amazingly simple way to grow your calendaring service exponentially. And if you get a manager that uses your calendar he might just tell everyone in a company to use your stuff and bam a lot of new users all ready to schedule weekend stuff with their friends, continuing the cycle of more users. Creating a community around your application is a good thing and is something that can get you laid.

If this seems familiar remember about a year ago now when Nat announced Hula. JWZ created a often referenced blog entry called Groupware bad telling how he was going to fail. The current set of calendars are better then Hula, but they still are missing the mark.

Of course I could be wrong and most people really do just want to have a calendar they themselves fill with things they are doing to view only by themselves, but I think that a much more likely answer is that calendaring sounds like a simple web2.0 company you could make and flip (how else would you come up with a horrible company name like 30boxes? And did you notice they have a gmail theme?). If that is your goal you don't think about growing users as fast as possible, but think about making a feature set that someone else would want to buy.

How much free space do you have?

2006-03-06T12:52:00.000-05:00

How much free space do you have is a pretty basic file manager issue that most linux file managers don't easily tell you. That is until today when I ran across Thunar today. Pretty slick.

Qt and Glib's event loop

2006-02-24T08:28:00.000-05:00

Over at the QDevBlog Brad posted about his efforts to integrate glib's event loop into Qt4. Fun work that I think you will all get a kick out of reading. "...I could write a plugin for the GIMP or for Eclipse, I could use the asynchronous GNOME VFS API..."