Writing a Custom Checker
The scoring logic for coding questions is relatively straight forward, i.e. making sure each test case result is an exact match. Some of the questions may have more than one acceptable correct answer. For example, "List of 5 prime numbers under 100"; therefore, such questions can be accommodated under Approximate Solution type.
When creating an Approximate Solution type question, the final step is writing a custom checker. A custom checker is a piece of code written by the problem setter to evaluate test case results and provide customized scoring logic.
Custom Checker Code
Custom checker code consists of 3 parts, two non-editable parts which are referred as head & tail section. Editable part of custom checker is referred as body section. Body section has a method named run_custom_checker(). This method takes two arguments, a TestStructure object, and a ResultStructure object. This method is executed after candidate's submission is run. The candidate's submission results are evaluated by each individual test case.
|testcase_id||[int] ID of the test-case|
[str] File path to test-case input
|testcase_output_path||[str] File path to test-case output generated by the problem solver|
|testcase_expected_output_path||[str] File path to test-case expected output to be matched with|
|metadata_file_paths||[list<str>] File paths to Question metadata (Extra files usually used for defining training sets)|
|submission_code_path||[str] File path to submission source code|
|testcase_result||[bool] Set to True if test-case output matches test-case expected output. Matching is done line by line|
[int] Exit code of the test-case process
|[float] Time taken by the test-case process in seconds|
|[int] Peak memory of the test-case process determined in bytes|
|[str] <Future use>|
|result||[bool] Assign test-case result. True determines success. False determines a failure|
[float] Assign test-case score. Normalized between 0 to 1
[str] Assign test-case message. This message is visible to the problem solver
Test case result is determined by setting values of ResultStructure object.
- result: Set this to True if a minimum cutoff score is achieved -- otherwise False. This decides whether you have cleared this test case or not.
- score: This is a normalized score, where 1 would fetch full score, and 0.5 would fetch half score.
- message: This is a customizable message which can be used to convey a message regarding test case result, e.g. “Failed because the cutoff was not reached."
Please don’t print anything to STDOUT in the run_custom_checker function.
Custom-Checker for list of 5 prime numbers under 100
import re # Function to determine if the given integer is a prime number def is_prime_number(x): if x >= 2: for y in range(2, x): if not x % y: return False else: return False return True def run_custom_checker(t_obj, r_obj): result_data = '' try: result_data = open(t_obj.testcase_output_path, 'r').read() except IOError: r_obj.result = False r_obj.score = 0 r_obj.message = 'Error reading result file' return # Read contents of the result file values = re.split(' |\n', result_data) # Make sure all the values are unique uniq_values = set(values) # Count the number of primes correct_values = 0 for value in uniq_values: if value and is_prime_number(int(value)): correct_values = correct_values + 1 # Cutoff score to determine success if correct_values > 3: r_obj.result = True else: r_obj.result = False r_obj.score = correct_values / 5 r_obj.message = str(correct_values) + ' out of 5 values are correct'
Here is a similar custom checker in Java - https://gist.github.com/patilarpith/7b17b5afb9218f770ebc528747017d25