Saturday, December 22, 2007

Valgrind callgrind tools Part 3: Code coverage

In the Valgrind manual they list some tools you could write for Valgrind. One of the suggestions is to write a coverage tool. So a few months ago I sat down and did exactly that. It was a nice little tool and I happily posted it to the Valgrind developer mailing list only to discover that I was not the first person to do this nor was any of our solutions that good in the end. The easy part of the project is obtaining a recored of what was executed. In fact Cachegrind does a fantastic job of that. The only thing my tool did (that I thought was new) was automatically collect data on all the jumps which discovered after I wrote my tool was already present in the callgrind tool if you pass in the argument "--collect-jumps=yes". So me and several others have all written bad versions of callgrind and the cg_annotate script. Code coverage with Valgrind is made up of two problems. The first is obtaining the code execution and the second is applying this over the source for the program. The first is easy by using "callgrind --collect-jump=yes". The second part isn't as easy a new problem for every language, but at least for C++ using Roberto's parser makes this task a lot easier.

Here is an older output for a test that I wrote for QColumnView that shows some missing coverage:

valgrind --tool=callgrind --collect-jumps=yes ./tst_qcolumnview

./callgrind_coverage ~/dev/qt/tests/auto/qcolumnview/callgrind.out.16089 ~/dev/qt/src/gui/itemviews/qcolumnview.cpp

Function coverage - Has each function in the program been executed?
Not executed: 612 void QColumnViewPrivate::_q_clicked(const QModelIndex &index)
Coverage: 39/40 97%

Condition coverage - Has each evaluation point (such as a true/false decision) been executed?
443 for (int i = 0; i < ranges; ++i) {
842 for (; (i < list.count() && i < d->columns.count()); ++i) {
212 if (d->columns.isEmpty() || dx == 0)
976 if (!model() || !selectionModel())
976 if (!model() || !selectionModel())
229 if (!index.isValid() || d->columns.isEmpty())
229 if (!index.isValid() || d->columns.isEmpty())
272 if (leftEdge > -horizontalOffset()
336 switch (cursorAction) {
338 if (current.parent().isValid() && current.parent() != rootIndex())
338 if (current.parent().isValid() && current.parent() != rootIndex())
557 for (int i = columns.size() - 1; i >= 0; --i) {
549 while (currentColumn == -1 && parentIndex.isValid()) {
572 if (currentColumn == -1 && parent.isValid())
577 if (build && columns.size() > currentColumn + 1) {
580 && !columns.at(currentColumn + 1)->rootIndex().isValid());
689 if (!columns.isEmpty() && columns.last()->isHidden())
689 if (!columns.isEmpty() && columns.last()->isHidden())
695 if (show && view->isHidden())
394 if (horizontalLength < viewportSize.width() && q->horizontalScrollBar()->value() == 0) {
813 if (!columns.isEmpty() && columns.last() == previewColumn)
1024 if (!model || columns.isEmpty())
1035 if (x != view->x() || viewportHeight != view->height())
1042 if (x != view->x() || viewportHeight != view->height())
368 if (diff < 0 && horizontalScrollBar()->isVisible()
877 if (currentParent == previous.parent()
877 if (currentParent == previous.parent()
879 for (int i = 0; i < d->columns.size(); ++i) {
893 for (int i = 0; i < d->columns.size(); ++i) {
512 if (!found && columns.at(i)->cornerWidget() == grip) {
Coverage: 566/598 377%

Statement coverage - Has each line of the source code been executed?
Coverage: 475/523 90%
-------------
Key:
c == compiled
e == executed
r == return
j == jump
s == skipped
- == comment/whitespace/not compiled
-------------
94 [j] f: }
273 [c]: && rightEdge <= ( -horizontalOffset() + viewport()->size().width())) {
286 [c]: newScrollbarValue = rightEdge + horizontalOffset();
336 [j] f: switch (cursorAction) {
369 [c]: && horizontalScrollBar()->value() == horizontalScrollBar()->maximum()) {
579 [c]: bool viewingChild = (!model->hasChildren(parent)
612 [c]: void QColumnViewPrivate::_q_clicked(const QModelIndex &index)
614 [c]: Q_Q(QColumnView);
615 [c]: QModelIndex parent = index.parent();
616 [c]: QAbstractItemView *columnClicked = 0;
617 [c]: for (int column = 0; column < columns.count(); ++column) {
618 [c]: if (columns.at(column)->rootIndex() == parent) {
619 [c]: columnClicked = columns[column];
623 [c]: if (q->selectionModel() && columnClicked) {
624 [c]: QItemSelectionModel::SelectionFlags flags = QItemSelectionModel::Current;
625 [c]: if (columnClicked->selectionModel()->isSelected(index))
626 [c]: flags |= QItemSelectionModel::Select;
627 [c]: q->selectionModel()->setCurrentIndex(index, flags);
690 [c]: columns.last()->setVisible(true);
726 [c]: model()->fetchMore(index);
824 [c]: previewColumn->setMinimumWidth(qMax(previewColumn->verticalScrollBar()->width(),
861 [rc]: return list;
878 [c]: && model()->hasChildren(current) && model()->hasChildren(previous)) {
939 [c]: QItemSelectionModel *replacementSelectionModel =
941 [c]: replacementSelectionModel->setCurrentIndex(
943 [c]: replacementSelectionModel->select(
993 [c]: QModelIndex br = model()->index(model()->rowCount(parent) - 1,
994 [c]: model()->columnCount(parent) - 1,
1081 [c]: opt.rect = QRect(option.rect.x() + option.rect.width() - width, option.rect.y(),


Note that this tool is a bit of a hack and is a work in progress so you will no doubt find issues. But it is good enough to be useful so I figured it would be time to publish:

The source for callgrind_coverage can be found in my callgrind tools git repository.

6 comments:

Kevin Kofler said...

"Coverage: 566/598 377%" - Huh? ;-)

Benjamin Meyer said...

Yup, that is one of the "issues" I was talking about. Getting the list of the actual missed functions/jumps/lines was more important to me then the % so I have neglected to fix that yet. But I have begun running a nightly cron so I will be fixing that bug this week.

renoX said...

I wonder if the time that you and other wasted reinventing what is already existing isn't an indicator that there's something missing in the documentation?

Maybe you could fix this, so that other don't do the same thing again..

Benjamin Meyer said...

yah the Valgrind docs should be tweaked to stop encouraging people to write coverage tools.

David Bar said...

Benjamin,

I realize it has been 8 years since your original post, but a few questions:
1) Is there an alternative project that achieves the same thing? I've searched, but couldn't find anything similar.
2) Is there a newer version of your code?
3) Any chance of having this code without Qt / rpp dependencies?

Benjamin Meyer said...

1) I am not aware of any alternatives for C/C++, but I have not recently looked around either
2) The latest version is what is found on the git repository.
3) The project is pretty small and it should be pretty easy to re-write using llvm if you wanted to. I personally have no incentive to as it works for what I need as is.

Popular Posts