Sunday, October 29, 2006

Netflix Prize Contest

I was very excited when the Netflix contest was first announced. I have been toying with that problem for a while, but I never had enough data to do anything really with so it was always a minor project of mine. As I watched people get interested in the contest many of them were spending the first week figuring out how to manage all the data that was given to them. The worst part was that after a week of doing this it wasn't very fun and they would stop, but yet they hadn't even worked on the real problem yet! I have been staying up late this week, but having a blast and created a little present for you all, the NetflixRecommenderFramework. It lets you very quickly start hacking on your algorithm ideas without having to bootstrap everything else first. It even includes autotests. Maybe you can win the million dollars.

On the Qt side of things one of the included tools is a small GUI application that can be used to explore the database. At over 100 million rows I am happy to say that Qt QTableView can handle it just fine and there is no startup delay. This is a followup to last years blog entry 66 million rows in QTreeView where I got it up to 66 million rows without any work. This time around I have actual data behind each row and column and not just a model that generates the data based upon the row and column. The bugs found before were still there if I set the height of the rows too large at which point their total height was greater then max int, but I have fixed those issues and they will be in the next Qt release. Anyone know if Gtk can do this?



Update: fixed link to project and source.

3 comments:

Hugo said...

Dear Benjamin Meyer,

You indicate that already in 2006 there would be a 'next Qt release' where the 'rows * rowheight > maxint' problem with QTableView would be fixed!

However, the problem seems to persist, e.g.: https://bugreports.qt-project.org/browse/QTBUG-18490

Could you inform us about what happened to this wonderful fix you mention?

Thanks!

Benjamin Meyer said...

@Hugo, sounds like there might be a regression in the QTableView between 4.6 and 4.8. At the time I wrote up some simple unit tests with a large model to catch most of the errors. As I have not worked on Qt in several years I don't know the current status of it. Running example applications against older versions of Qt would confirm/deny if it is a regression.

Hugo said...

Thank you Benjamin, trying with Qt 4.6 will be a good start!

Popular Posts