Overview
HackerRank Projects for Data Science allows you to set up automatic scoring for your custom project-based real-world questions to assess Data Scientists.
This is an optional step under the Setup Project step of the Data Science question creation workflow that allows you to set up automatic scoring. You can add evaluation scripts, submission files, solution notebooks, and so on in the section Add evaluation files. All files uploaded in this field will be hidden and unavailable to the candidate. These files will be downloaded in the Juypter session during scoring and evaluation inside the "evaluation_files" folder. The configuration file Hackerrank.yml needs to have the scoring command. The scoring command would be executed automatically after a candidate submits the solution.
In this article, we will walk you through the process of setting up automatic scoring while creating your own Data Science Question on HackerRank Projects.
Setting Up Automatic Scoring of Data Science Questions
There are two options to set up scoring
Standard Metrics
Select a metric that you would like to use to evaluate your candidate’s submission.
Add the file name which you would ask your candidate to submit their final submission.
Upload the actual output file containing the result, which will be used to generate the score and Map the required fields for scoring.
Upload the expected submission file that you expect the candidate to submit and Map the required fields for scoring. This file will be used to validate the scoring setup.
Click on the validate and save buttons to validate the changes. In case of errors, follow error logs and messages to debug.
Custom scoring
- Create a program to evaluate candidate submissions.
(Example: a python program that will compare the candidate's submission.csv with the uploaded actual_output.csv and calculate the desired model performance metric such as accuracy, recall, precision, f-score, etc., convert that into a % and return FS_SCORE: X% as output)
- Ensure the scoring program returns in the format “FS_SCORE: X%.”
- Upload all the necessary files (actual_output.csv) needed to run the evaluation program successfully, along with the scoring program(score.py, score.sh), in the evaluation section.
- Edit the hackerrank.yml configuration file and add the scoring key with the scoring command. The example configuration is below:
Scoring
command: python evaluation_files/score.py evaluation_files/actual_output.csv
- Ensure that your evaluation script can handle edge cases and will return a score in all possible scenarios.
- An erroneous response will result in a no score, and you will be required to review the submission manually.
- Click on the validate and save buttons to validate the changes. In case of errors, follow error logs and messages to debug.
Note: To update the evaluation files, you must re-upload them and delete the previous version.
HackerRank also offers manual scoring for the Data Science questions. Learn more about it here.