AI-Powered Plagiarism Detection

Overview

Preventing plagiarism in online assessments has always been important. The integrity of an assessment also relies on plagiarism detection.

With the widespread availability of AI tools such as ChatGPT, there is a rising need for plagiarism strategies to ensure all developers get an opportunity to demonstrate their unique skill sets.

Note: The Plagiarism detection is available only for the Coding type questions in HackerRank. 

Why AI-Powered Plagiarism Detection?

Code similarity has been the primary signal for plagiarism detection across the online coding assessment industry. The traditional code similarity-based approach caused a significant false positive rate of (70-80%) has been observed when coding questions tend to have limited solution patterns. This false positive rate requires more manual reviews by administrators. Therefore, the main goal of the AI-powered plagiarism solution is to decrease the false positive rate and protect candidates from being incorrectly flagged for plagiarism.

Generative AI tools such as ChatGPT are complex for a code similarity-based detection method because a new response could be generated for each user. The AI-powered solution will be able to flag such cases of cheating reliably.

How HackerRank Detects Plagiarism with AI

The new plagiarism detection system feeds several proctoring and user-generated signals into a supervised Machine Learning algorithm to flag suspicious behavior during an assessment. By understanding code iterations made by the candidate, the model can detect if they had external help, including from ChatGPT. The result is a solution with an AI-based plagiarism detection system with a 93% accuracy rate, dramatically reducing false positives from the traditional model.

Besides reducing false positives, the new AI-powered model improves precision and recall over traditional Moss code similarity models.

The table describes a comparison of the Precision and Recall values for the AI-powered and Moss Code similarity models:

The following table describes some of the key differences between HackerRank's AI-powered model and the Moss Code similarity model:

Plagiarism Flag

HackerRank's plagiarism flag indicates that someone may be submitting code originating from another source.

Even with 93% accuracy, we recommend that a developer should review the code in the playback to decide if this is an actual case of plagiarism or not. It is not recommended to auto-reject a candidate based on the plagiarism flag.
We offer a keystroke-by-keystroke Codeplayer feature within the HackerRank product, enabling customers to thoroughly review and verify plagiarism prediction results when incorporating their feedback. Additionally, this information can also be verified on the CSV report that the customers can download from the platform.
For further information, see How to Check for Suspicious Activity Using the AI-Powered Plagiarism Detection Solution. HackerRank ensures that the solution should be candidate first and candidates are not penalized unnecessarily, and at the same time, suspicious candidates are captured reliably.

Be aware:

• The Codeplayer will only show the most recent language used by the candidate for submitting a solution to a question.
• If questions do not have the playback option available, HackerRank recommends you refer to other signals available on the Candidate report and additional signals in the CSV report to decide on the candidate attempt.

How to Enable AI-Powered Plagiarism Detection for Your Tests?

AI-Powered Plagiarism is turned off by default. You can enable the feature after consulting with your team.

1. Navigate to the Test Settings of a Test.
2. Click the Test Integrity tab.
3. Under Plagiarism, enable the AI Plagiarism Detection toggle.
4. You can click on the help icon to view the related support article. You can also view the HackerRank NYC laws for AI in the article FAQ section of HackerRank Services and AI Laws, by clicking the link here.
5. Once you enable AI-powered plagiarism detection, the following prompt displays the effects on your workflow:

• • • Enabling plagiarism detection may cause some existing macros to fail
    • You can view the AI-powered plagiarism detection on the Detailed Report 

5. When you click Agree & Enable, the AI-powered plagiarism model will be used to detect any suspicious activity of a Candidate.

For information on detecting suspicious activity on Tests with the AI-powered Plagiarism Detection solution, see the article How to Check for Suspicious Activity Using the AI-Powered Plagiarism Detection Solution.

Candidate Experience

When you enable AI plagiarism detection, the system automatically takes a Candidate's consent to proceed.

1. When the Candidate launches the Test, a prompt message is displayed to provide consent that the Candidate is fine with taking the test that has artificial intelligence analysis.
   
2. The Candidate can agree to the declaration and click Continue to take the Test.

Can the AI-Powered Solution Detect Cheating Using ChatGPT?

Consider a solution written by a candidate with the help of ChatGPT and how AI-powered plagiarism detection identifies it.

In the following screenshot, the candidate tries to bypass the MOSS similarity check by prompting ChatGPT to provide a unique solution:

Additionally, the candidate can request ChatGPT to provide a solution that is very long and less common:

The candidate can also convert the solution into niche programming languages:

These techniques help candidates bypass the code similarity check quickly. 

However, when the same solution was passed through HackerRank’s AI-powered plagiarism detection solution, the solution captured the attempt as suspicious with a suspicion score of 9 (which is considered ‘Very High’). Therefore, you can be rest assured that candidates resorting to cheating using ChatGPT will now be caught with HackerRank’s AI-powered plagiarism detection solution.

This solution was plagiarized on top of a solution generated by ChatGPT. Even though the candidate did not use direct copy-paste, the HackerRank AI model successfully marked it as highly suspicious.

From the below screenshot, you can view that our model identifies plagiarism due to coding patterns. Users can review this and confirm it by using the feature that replays code input keystroke by keystroke. Note that there is no code similarity indicator with this attempt, which means that this attempt passed the code similarity check easily.

Note: Use of HackerRank's AI-powered plagiarism solution may be subject to the NYC Automated Employment Decision Tool Law to the extent a user uses the solution as an “automated employment decision tool” within New York City. HackerRank has incorporated certain functionality within the solution to assist the users with compliance based on the NYC Law.

For more information on HackerRank's NYC AI-powered plagiarism, see the article FAQ section of HackerRank Services and AI Laws. Additionally, HackerRank's AI-powered plagiarism solution has gone through an independent bias audit consistent with the NYC AI Law. Summary of Bias Audit Results of the HackerRank's Plagiarism Detection System for New York City's Local Law 144

Limitations of AI-powered Plagiarism Detection Model

The HackerRank AI-powered plagiarism model does not assess for plagiarism if a question requires limited effort to receive a full score or if the candidates’ solution contains very few lines of code. Such questions typically lack adequate indications for reliable detection; however, we still flag the response for potential plagiarism if substantial external copy-pasting is identified for those questions. 

Important: As the demand for generative large language models increases, it is recommended that users include at least one question of Medium difficulty level in the Test.