Creating a Custom Checker – HackerRank Support Center

Hide

In this article

Overview

The scoring logic for coding questions is relatively straightforward, i.e., making sure each test case result is an exact match. Some of the questions may have more than one acceptable correct answer. For example, "List of 5 prime numbers under 100"; therefore, such questions can be accommodated under the Approximate Solution type.

When creating an Approximate Solution type question, the final step is writing a custom checker. A custom checker is a code written by the problem setter to evaluate test case results and provide customized scoring logic.

Custom Checker Code

Custom checker code consists of 3 parts. Two non-editable parts are referred to as a head & tail section. The editable part of the custom checker is referred to as the body section. Body section has a method named run_custom_checker(). This method takes two arguments, a TestStructure object and a ResultStructure object. This method is executed after a candidate's submission is run. Each individual test case evaluates the candidate's submission results.

TestStructure::

testcase_id	[int] ID of the test-case
testcase_input_path	[str] File path to test-case input
testcase_output_path	[str] File path to test-case output generated by the problem solver
testcase_expected_output_path	[str] File path to test-case expected output to be matched with
metadata_file_paths	[list<str>] File paths to Question metadata (Extra files usually used for defining training sets)
submission_code_path	[str] File path to submission source code
testcase_result	[bool] Set to True if test-case output matches test-case expected output. Matching is done line by line
testcase_signal	[int] Exit code of the test-case process
testcase_time	[float] Time is taken by the test-case process in seconds
testcase_memory	[int] Peak memory of the test-case process determined in bytes
data	[str] <Future use>

ResultStructure::

result	[bool] Assign test-case result. True determines success. False determines a failure
score	[float] Assign test-case score. Normalized between 0 to 1
message	[str] Assign test-case message. This message is visible to the problem solver.

Test case result is determined by setting values of ResultStructure object.

Result: Set this to True if a minimum cutoff score is achieved -- otherwise, False. This decides whether you have cleared this test case or not.
Score: This is a normalized score, where one would fetch the total score, and 0.5 would fetch half score.
message: This is a customizable message which can be used to convey a message regarding a test case result, e.g., “Failed because the cutoff was not reached."

Note: Please don’t print anything to STDOUT in the run_custom_checker function.

Custom-Checker for list of 5 prime numbers under 100

import re
# Function to determine if the given integer is a prime number
def is_prime_number(x):
    if x >= 2:
        for y in range(2, x):
            if not x % y:
                return False
    else:
        return False
    return True

def run_custom_checker(t_obj, r_obj):
    result_data = ''
    try:
        result_data = open(t_obj.testcase_output_path, 'r').read()
    except IOError:
        r_obj.result = False
        r_obj.score = 0
        r_obj.message = 'Error reading result file'
        return

    # Read contents of the result file
    values = re.split(' |\n', result_data)

    # Make sure all the values are unique
    uniq_values = set(values)

    # Count the number of primes
    correct_values = 0
    for value in uniq_values:
        if value and is_prime_number(int(value)):
                correct_values = correct_values + 1

    # Cutoff score to determine success
    if correct_values > 3:
        r_obj.result = True
    else:
        r_obj.result = False
    r_obj.score = correct_values / 5
    r_obj.message = str(correct_values) + ' out of 5 values are correct'

Result

Here is a similar custom checker in Java - https://gist.github.com/patilarpith/7b17b5afb9218f770ebc528747017d25