Better feedback.
Better sutdents.

BeyondGrader evaluates multiple facets of code quality—style and formatting, cyclomatic complexity, execution efficiency, source line counts, memory usage, and dead code detection.

We can help you teach students to create code that is not just correct, but also beautiful, efficient, and idiomatic. Let’s see how!

Example Problem

The examples below use the following example problem. Toward the end, we’ll show how easy it is to write this question!

Feel free to give it a try! We won’t repeat the instructions below.

Array Squares Sorted

Geoffrey Challen // 2022.3.0

Create a class SquaredSorter that provides a single class method named maxOfSortedSquares. maxOfSortedSquares accepts an array of ints. If the passed array is null or empty, throw an IllegalArgumentException.

Otherwise, proceed as follows. First, square every value of the array. Next, sort the array in ascending order. Finally, return the maximum of the squared, sorted array. Modify the array in place, and do not create a copy. Sort the array using a helper method from java.util.Arrays.

// Write your solution here...

Correctness

Before we discuss code quality, we don’t want you to miss the fact that BeyondGrader is also a fantastic autograder. We have designed the feedback to be clear, concise, and useful to beginning programmers.

Check out the walkthrough below to find out more!

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    for (int i = 0; i < values.length; i++) {
      values[i] = values[i] * values[i];
    }
    return values[values.length - 1];
  }
}

Style and Formatting

What use is correct code if it’s unreadable? In our experience, teaching students to naturally write code to a style guide is extremely valuable. Certain aspects of style—such as whitespace and indentation—have a huge impact on intelligibility and a student’s ability to understand their own code. And, when all your students are creating consistently-formatted code, it’s much easier for staff to read their code when helping them.

BeyondGrader implements the Sun Java style guide using checkstyle. This is the most commonly-used set of formatting rules for Java, and a superset of Google’s guidelines. For Kotlin we use ktlint and it’s default settings. checkstyle provides helpful error messages, and students quickly become used to its conventions.

The walkthrough below discusses various aspects of style and formatting examined by BeyondGrader.

import java.util.Arrays;

public class SquaredSorter {
  static public int maxOfSortedSquares (int[] values)
  {
if (values == null || values.length==0) {
  throw
  new IllegalArgumentException();
}
    for (
      int i = 0; i < values.length; i++
    ) {
      values[i] = values[i]*values[i];
    }
    Arrays.sort(values); return values[values.length - 1];
  }}

Cyclomatic Complexity

Cyclomatic complexity measures how many different ways control can flow through a program. In our experience, extra code paths in student code typically reflect a misunderstanding about how to correctly approach the problem. BeyondGrader will flag submissions that add too many code paths when compared to the reference solution.

Let’s examine an example of overly-complicated code in the walkthrough below.

import java.util.Arrays;

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    if (values == null || values.length == 0) {
      throw new IllegalArgumentException();
    }
    for (int i = 0; i < values.length; i++) {
      if (values[i] > 0) {
        values[i] = values[i] * values[i];
      } else if (values[i] == 0) {
        values[i] = 0;
      } else {
        values[i] = (-1 * values[i]) * (-1 * values[i]);
      }
    }
    Arrays.sort(values);
    return values[values.length - 1];
  }
}

Execution Efficiency

Sometimes it’s appropriate to stress performance when working with students. Sometimes not. Regardless, extreme inefficiency almost always points to deeper problems with a submission.

BeyondGrader uses bytecode instrumentation to measure the number of lines executed by a submission. This is a stable metric—unlike wall clock time—and lets you establish tight efficiency bounds when appropriate⁽¹⁾. When the execution line count exceeds the limit, students are prompted to improve their code.

In the walkthrough below, we examine an example of how evaluating this aspect of code quality produces better submissions.

import java.util.Arrays;

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    if (values == null || values.length == 0) {
      throw new IllegalArgumentException();
    }
    for (int i = 0; i < values.length; i++) {
      for (int j = 0; j < 2; j++) {
        int value = values[i] * values[i];
      }
      values[i] = values[i] * values[i];
    }
    Arrays.sort(values);
    return values[values.length - 1];
  }
}

Source Line Counts

Setting reasonable limits on the number of lines in a submission can flag common programming errors and student misconceptions⁽²⁾. Note that this is possible in BeyondGrader given brace-based languages like Java and Kotlin, due to the fact that our linting rules prevent students from collapsing everything down to a single line.

The example below exhibits a common student programming misconception, and the interactive walkthrough discusses how we can detect it using source line counts.

import java.util.Arrays;

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    if (values == null || values.length == 0) {
      IllegalArgumentException e = new IllegalArgumentException();
      throw e;
    }
    for (int i = 0; i < values.length; i++) {
      int value = values[i];
      int valueSquared = value * value;
      values[i] = valueSquared;
    }
    Arrays.sort(values);
    int max = values[values.length - 1];
    return max;
  }
}

Dead Code Detection

Portions of student submissions not covered during testing are very likely to indicate a mistake or a misunderstanding of how to solve the problem. So, while BeyondGrader by default will accept submissions that have cyclomatic and runtime complexity greater than the reference solution, it always flags any instance of unexecuted code.

import java.util.Arrays;

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    if (values == null || values.length == 0) {
      throw new IllegalArgumentException();
    }
    for (int i = 0; i < values.length; i++) {
      values[i] = values[i] * values[i];
    }
    Arrays.sort(values);
    if (values.length == 0) {
      return 0;
    }
    return values[values.length - 1];
  }
}

Memory Allocation Measurement

Regardless of whether you stress performance in your course, student submissions that allocate way too much memory can probably be improved. BeyondGrader combines bytecode rewriting with a custom Java agent to track and report memory allocation totals by student submissions, and provide feedback when memory usage has gotten out of hand.

import java.util.Arrays;

public class SquaredSorter {
  public static int maxOfSortedSquares(final int[] values) {
    if (values == null || values.length == 0) {
      throw new IllegalArgumentException();
    }
    for (int i = 0; i < values.length; i++) {
      values[i] = values[i] * values[i];
    }
    int[] valuesCopy = Arrays.copyOf(values, values.length);
    Arrays.sort(values);
    return values[values.length - 1];
  }
}

Coming Soon

We have several new exciting code quality features planned for upcoming releases:

Syntactic feature comparison: Should the submission use a loop, or maybe not if recursion was called for? Is the student declaring a bunch of extra variables? We’ll compare the language features used by the student submission to the reference solution to guide students towards more idiomatic solutions.
Complexity categorization: If an O(n) algorithm was expected, and the student submits something that’s O(n²), we should probably talk.
Variable name analysis: Tired of seeing foo, bar, and baz? So are we! We’ll help verify that student submissions are using meaningful variable names appropriate to the problem they are solving.

Problem Authoring

BeyondGrader goes beyond just providing code quality feedback. It also provides a framework for question development that makes writing accurate new problems fast, easy, and—maybe, just maybe—even fun! Notably, there is no need and no way to write test cases!

In the video we demonstrate the process of writing the question we’ve been experimenting with above. Check it out!

Better feedback.Better sutdents.

Array Squares Sorted

Better feedback.
Better sutdents.