Overview
The scoring logic for coding questions is relatively straightforward, i.e., making sure each test case result is an exact match. Some of the questions may have more than one acceptable correct answer. For example, "List of 5 prime numbers under 100"; therefore, such questions can be accommodated under the Approximate Solution type.
When creating an Approximate Solution type question, the final step is writing a custom checker. A custom checker is a code written by the problem setter to evaluate test case results and provide customized scoring logic.
Custom Checker Code
Custom checker code consists of 3 parts. Two non-editable parts are referred to as a head & tail section. The editable part of the custom checker is referred to as the body section. Body section has a method named run_custom_checker(). This method takes two arguments, a TestStructure object and a ResultStructure object. This method is executed after a candidate's submission is run. Each individual test case evaluates the candidate's submission results.
TestStructure::
testcase_id | [int] ID of the test-case |
testcase_input_path |
[str] File path to test-case input |
testcase_output_path | [str] File path to test-case output generated by the problem solver |
testcase_expected_output_path | [str] File path to test-case expected output to be matched with |
metadata_file_paths | [list<str>] File paths to Question metadata (Extra files usually used for defining training sets) |
submission_code_path | [str] File path to submission source code |
testcase_result | [bool] Set to True if test-case output matches test-case expected output. Matching is done line by line |
testcase_signal |
[int] Exit code of the test-case process |
testcase_time |
[float] Time is taken by the test-case process in seconds |
testcase_memory |
[int] Peak memory of the test-case process determined in bytes |
data |
[str] <Future use> |
ResultStructure::
result | [bool] Assign test-case result. True determines success. False determines a failure |
score |
[float] Assign test-case score. Normalized between 0 to 1 |
message |
[str] Assign test-case message. This message is visible to the problem solver. |
Test case result is determined by setting values of ResultStructure object.
- Result: Set this to True if a minimum cutoff score is achieved -- otherwise, False. This decides whether you have cleared this test case or not.
- Score: This is a normalized score, where one would fetch the total score, and 0.5 would fetch half score.
- message: This is a customizable message which can be used to convey a message regarding a test case result, e.g., “Failed because the cutoff was not reached."
Note: Please don’t print anything to STDOUT in the run_custom_checker function.
Custom-Checker for list of 5 prime numbers under 100
import re # Function to determine if the given integer is a prime number def is_prime_number(x): if x >= 2: for y in range(2, x): if not x % y: return False else: return False return True def run_custom_checker(t_obj, r_obj): result_data = '' try: result_data = open(t_obj.testcase_output_path, 'r').read() except IOError: r_obj.result = False r_obj.score = 0 r_obj.message = 'Error reading result file' return # Read contents of the result file values = re.split(' |\n', result_data) # Make sure all the values are unique uniq_values = set(values) # Count the number of primes correct_values = 0 for value in uniq_values: if value and is_prime_number(int(value)): correct_values = correct_values + 1 # Cutoff score to determine success if correct_values > 3: r_obj.result = True else: r_obj.result = False r_obj.score = correct_values / 5 r_obj.message = str(correct_values) + ' out of 5 values are correct'
Result
Here is a similar custom checker in Java - https://gist.github.com/patilarpith/7b17b5afb9218f770ebc528747017d25