Skip to main content
↑

Creating a Custom Checker

  • Updated
Hide
In this article

Custom Checker Code

Custom checker code consists of 3 parts. Two non-editable parts are referred to as a head & tail section. The editable part of the custom checker is referred to as the body section. The body section has a method named run_custom_checker(). This method takes two arguments, a TestStructure object and a ResultStructure object. This method is executed after a candidate's submission is run. Each test case evaluates the candidate's submission results.

custom_checker.jpg

Test Structure

testcase_id [int] ID of the test-case
testcase_input_path

[str] File path to test-case input
testcase_output_path [str] File path to test-case output generated by the problem solver
testcase_expected_output_path [str] File path to test-case expected output to be matched with
metadata_file_paths [list<str>] File paths to Question metadata (Extra files usually used for defining training sets)
submission_code_path [str] File path to submission source code
testcase_result [bool] Set to True if test-case output matches test-case expected output. Matching is done line by line

testcase_signal

[int] Exit code of the test-case process

testcase_time

 [float] Time is taken by the test-case process in seconds

testcase_memory

 [int] Peak memory of the test-case process determined in bytes

data

 [str] <Future use>

Result Structure

result [bool]  Assign test-case result. True determines success. False determines a failure
score

[float] Assign test-case score. Normalized between 0 to 1
message

[str] Assign test-case message. This message is visible to the problem solver.

 Test case result is determined by setting values of ResultStructure object.

  1. Result: Set this to True if a minimum cutoff score is achieved, otherwise, False. This decides whether you have cleared this test case or not.
  2. Score: This is a normalized score, where one would fetch the total score, and 0.5 would fetch half the score.
  3. Message: This is a customizable message that can be used to convey a message regarding a test case result, e.g., “Failed because the cutoff was not reached."

Note: Please don’t print anything to STDOUT in the run_custom_checker function. 

Custom-Checker for list of 5 prime numbers under 100 

import re
# Function to determine if the given integer is a prime number
def is_prime_number(x):
    if x >= 2:
        for y in range(2, x):
            if not x % y:
                return False
    else:
        return False
    return True

def run_custom_checker(t_obj, r_obj):
    result_data = ''
    try:
        result_data = open(t_obj.testcase_output_path, 'r').read()
    except IOError:
        r_obj.result = False
        r_obj.score = 0
        r_obj.message = 'Error reading result file'
        return

    # Read contents of the result file
    values = re.split(' |\n', result_data)

    # Make sure all the values are unique
    uniq_values = set(values)

    # Count the number of primes
    correct_values = 0
    for value in uniq_values:
        if value and is_prime_number(int(value)):
                correct_values = correct_values + 1

    # Cutoff score to determine success
    if correct_values > 3:
        r_obj.result = True
    else:
        r_obj.result = False
    r_obj.score = correct_values / 5
    r_obj.message = str(correct_values) + ' out of 5 values are correct'

Result

 Here is a similar custom checker in Java - https://gist.github.com/patilarpith/7b17b5afb9218f770ebc528747017d25

Related articles