CASET: Complexity Analysis using Simple Execution Traces for CS* submissions

TL;DR

CASET is a novel tool that uses execution traces and machine learning to analyze and classify the time complexity of student algorithms, aiding automated grading and feedback in CS education.

Contribution

The paper introduces CASET, a new method combining dynamic execution traces and unsupervised learning to determine algorithm complexity without source code analysis.

Findings

Successfully classifies sorting, searching, and dynamic programming algorithms.

Helps tutors identify algorithms used without reading source code.

Enhances automated grading with complexity analysis.

Abstract

The most common method to auto-grade a student's submission in a CS1 or a CS2 course is to run it against a pre-defined test suite and compare the results against reference results. However, this technique cannot be used if the correctness of the solution goes beyond simple output, such as the algorithm used to obtain the result. There is no convenient method for the graders to identify the kind of algorithm used in solving a problem. They must read the source code and understand the algorithm implemented and its features, which makes the process tedious. We propose CASET(Complexity Analysis using Simple Execution Traces), a novel tool to analyze the time complexity of algorithms using dynamic traces and unsupervised machine learning. CASET makes it convenient for tutors to classify the submissions for a program into time complexity baskets. Thus, tutors can identify the algorithms used by the submissions without necessarily going through the code written by the students. CASET's analysis can be used to improve grading and provide detailed feedback for submissions that try to match the results without a proper algorithm, for example, hard-coding a binary result, pattern-matching the visible or common inputs. We show the effectiveness of CASET by computing the time complexity of many classes of algorithms like sorting, searching and those using dynamic programming paradigm.