Friday, October 17, 2008

Git Hooks

Git hooks are scripts that are run by Git before or after certain commands. Because the hooks are run locally and not on the server it allows for a lot of freedom to write more in depth scripts. The scripts are located in the .git/hooks directory and Git comes with some sample scripts that can be enable by removing the .sample suffix and making it executable.

A nice example is the included pre-commit script which checks for trailing whitespace in the patch and exits with 1 if any are found. If a script exits with non-zero then the command is aborted.

Below is a simple, but very useful script. All it does is run make in your current directory if a Makefile exists. If make fails then it returns non zero preventing the commit from occurring. While this script is crude and verbose, it works in preventing commits the break the build. Compared to a server side script which has to build the entire project from scratch on potentially a slow machine, there should be nothing to build (you did run make before committing right?) so it should return very quickly. With this one script you can prevent most build errors from getting pushed up to the central repository and causing problems for other developers. One liner "fixes" that break the build can be a thing of the past.

pre-commit_make

#!/bin/sh
echo "--Attempting to build--"
if [ -f Makefile ] ; then
make --quiet
if [ $? != 0 ] ; then
echo "--Build failure--";
exit 1
fi
fi
echo "--Attempting to build pass--"
And really you can simplify it down to just this
#!/bin/sh
[ ! -f Makefile ] || make


Files in the .git/hooks directory are not part of the repository and so they are not tracked. A workaround is to have a git_hooks directory at the top of your repository like done in Arora and symlink .git/hooks to git_hooks whenever you clone. This way the hooks will be part of the project, under revision control and accessible to everyone.

When using git commit if you want to skip the hooks you can pass the --no-verify option.

Although many hooks you see are written in shell script they can be in any language don't even need to be a script, just an executable in the hooks directory.

An important thing to remember when writing hooks is that the patch and the list of patched files is available. A script that validates XML shouldn't do anything if the change doesn't modify xml files. 'git diff-index --name-only HEAD' is one simple way to get the list of all the changed files Using this you can make sure that no the hooks will always run quickly and only test/check what is needed.

There are a number of different hooks that git provides access to. For a complete list and details about what each of them are for checkout the Git hooks documentation.

Here are some hooks that I have either thought about doing, have already done, or found via Google:

pre-commit
- Check that the project compiles
- Check for compiler warnings
- Check that the auto tests for the files that are being modified still pass.
- Check that the performance tests for the files that are being modified still pass.
- Check that the code in the patch conforms to a code style
- Validate XML files
- Spell check user strings, documentation, etc
- Check for binary incompatibility
- Try to build a release package
- Check that any public API is documented and has no errors
- Verify any new files have a copyright header
- Check that all public strings are properly wrapped so they can be translated
- Check for calls to functions that are known to be deprecated or inefficient.
- Check for swear words
- Reject commits made between 4am and 7am with a note to go to bed.
- Automatically remove whitespace
- Run RSpec tests before allowing a commit
- Some rake testing

commit-msg
- Spell check the commit message
- Verify an agreed upon commit message format is used.
- Prevent business announcements or key words from leaking out in a message.

pre-rebase
- Useful for preventing an accidental rebase from occurring on a master branch that everyone shares.

post-commit
- Send out an e-mail to the mailing list
- Build a package
- Build/Update API documentation for the website
- Update/close bugs in a bug tracking system
- Spawn off builds on other operating systems

post-receive
- Update tickets in Lighthouse

I have mostly played around with the user side where each commit is one patch and life is pretty simple, but on the server a push can contain a number of commits and the scripts are only called once. Checking out the docs and the scripts in the git source code contrib/hooks directory for more information on how to extract each commit.

Taking advantage of the hooks system one can easily add checks for all sorts of things that can be automated resulting in always building source that is higher quality, more consistent, and with less embarrassing errors. The ease that anyone can add hooks will hopefully cause more and more projects to utilize this built in part of git.

If you have written an interesting hook feel free to post a link to it in the comments for others to checkout.

Hook photo taken by *L*u*z*a*

Tuesday, October 07, 2008

LSDVD

LSDVD was the name of a successful project to create a fully functional software DVD player for Linux back in 1999.

At the start of my freshman year at RIT in 1999 A guy by the name of Gad gave a presentation about the DVD format and his project to create a software DVD player for Linux. This was all back before DeCSS, nearly every DVD related tool that exists today and really before Linux was used by many as a desktop machine. I didn't even own a DVD reader for my computer, but it sounded like an exciting project so I picked up a 2X dvd drive for $109 and ordered my first DVD online.

I got involved in the project by working on the graphical front-end for the application. Over time I worked on many areas of the project, but I was in charge of the front-end from then on. We worked on and off on the project while going to school and through the winter, but the project was going very slowly. RIT runs on the quarter system and so in the spring took time off and rented a house north of RIT in Greece so we could work on the project full time. Day in and day out we worked on the project. Moving into the house with only a few hundred dollars to my name I slept on a couch in the basement and ate what canned food and Ramen noodles I had. We bought wood from Home Depot and built ourselves some desks for our computers and placed them in the living rooms. The house was pretty much empty other then a kitchen table and a mattress in one of the rooms. We would wake up, sit at the computer and code till it was time to sleep. Highlights included when someone would come over and we would have a real cooked meal (and watching office space).

Before DeCSS there was very little selection for DVD's to test because we could only read DVD's that were not encrypted and region free. This turned out to be mostly old gore flicks. It was a nice welcome when we could finally access our DVD collection to test against. The other two guys in the group were Paul and Dave. Dave was a crazy hacker who was in charge of the video decoding. Paul was in charge of the project. He set us up with the cvs server, managed the build script, all the small things that a project needs to survive, and of course huge chunks of the DVD player itself.

When we first got the MPEG2 decoder written it was still slow and so it would que twenty seconds of frames and then beep several times before playing it. We had a computer in the corner of the room playing Enemy of the State. Every five minutes it would beep and we would all run over to watch the video play. It took at least a day or so to get through the movie.

At each stage there was a fun magical moment as it came together for the first time. Getting audio working for the first time. Getting the audio and video sync working, subtitles, menus, and more. And of course we had to do testing too :)

Gad was looking for a company to support us and luckily one decided to buy us. Two months after we had moved into the house in Greece, we were acquired by a company in California. The four of us had never officially formed a company in that short time and so in trade for the code we got a small sign on bonus. Part of the agreement was that we had to live in California (which they paid for) and we were all excited about that. Flying out to California we spent the summer working on the DVD player. I have many good memories from that trip. We got to see a bit of California and went to a number of trade shows. I really grew as a programmer on this project, learning quite a bit from the other three especially Paul who designed the system. It was my first real large-scale project. Returning to Rochester that fall we continued to work on the player and got it to almost release status, but alas the project was killed in April 2001 due to financial problems within the company. It was less then a year later when the company was sold. The only one who have the application now is a small server sitting somewhere in the new company with our CVS code repository on it.

LSDVD includes the following features
* Almost perfect (within 1 frames) audio/video synchronization.
* Ran smoothly on a Celeron300a.
* Full menu support including subpictures.
* Graphical frontend.
* Didn't need to mount the media before playing.
* Worked on SMP machines.
* Fully supported AC3/PCM audio.
* All of the features that you would expect a stand alone DVD player to have such as seamless branches.

Whenever I see how little the software dvd players on Linux can still do and how much CPU they require I always have to smile.

Below is the one screenshot I still have of the application which was our easter egg and team photo.

My Kipling hacker bag

In March of 1999 the Belgian bag manufacturing company Kipling created a contest to promote their new line of "Hacker" clothes. The first 99 people to guess the login and password to their contest site got a "Hacker" bag. This was an interesting enough of a story to get it posted on http://www.slashdot.org which is where I saw it. I began to work on the problem and with the help of several others eventually got a bag.

The way that you found out if your login and password was correct was by a JavaScript program embedded in the web page. Having done some JavaScript the previous year I figured that it would be pretty easy to modify the page and JavaScript code to go through every possible combination until the winner was found.

Pretty soon I knew that the JavaScript engines of the day would be way to slow and a C/C++ version was needed. Later that afternoon when I got a version working I posted it on the discussion forums on Slashdot. Several other people were interested and began to help improve the application. A whole week with little sleep went by while we spend time optimizing the code, making sure the code was creating correct output, calculating how many years it would take to complete, and chatting on IRC.

By the middle of the week not much had changed. On the website they hinted that the username was 16 character long and the password was 4 characters long. The backpack that you would win was called "Host" and so we were almost sure that was the password. We planned a couple of wild guesses that might give us the answer for the login. The first idea was that the subscript on the packs was edgyhipunique or something like that. Although it was 16 characters, it was not the solution. So we went further. Next we tried all combinations of the first 16 letters of this sentence. All combinations were checked out, but nothing was found there either.

As we were continuing to make speed improvements a web site was set up to track what blocks people where checking so that people wouldn't check the same block twice. In all over 500 people were helping out in checking for the answer.

In the end someone went to one of Kipling's stores and figured out that one of the bags ("mailbomb") serial number appended with 001 was in fact the login.
Login: 9840112000309001
Password: host
In the end the brute force solution didn't find the login and password, but it was fun and I submitted the answer anyway. To my surprise a few months later got a bag in the mail. While the bag was neat it was the thrill of hacking on the code that I enjoyed much more.

I found the source tarball on a backup CD and have put it up online if anyone cares to look. It includes some binaries which probably don't work anymore and some messing around with the javscript too. Looking at the code I am horrified at it, but I was very happy that when I ran "./a.out -login 9840112000309001" it was quickly able to find the winning combination.

Popular Posts