Summary of Bias Audit Results of the HackerRank's Plagiarism Detection System for New York City's Local Law 144

Letter from the Lead Auditor

From: Shea Brown
           Lead Auditor
           BABL AI Inc.
           sheabrown@babl.ai

To: Interviewstreet Incorporation (HackerRank)
      PO Box 1660
      211 Hope Street
      Mountain View, California 94041

Re: Audit Opinion on HackerRank’s Plagiarism Detection System

07/22/2024

We have independently audited the bias testing assertions and related documentary evidence of HackerRank (the "Company") as of 07/22/2024, presented to BABL AI in relation to Company’s Plagiarism Detection System (the “system”) in accordance with the criteria and audit methodology set forth in this report. The goals of this audit are to:

1. Determine whether the bias testing methodologies, controls, and procedures performed by Company satisfy the audit criteria (see Findings)

2. Obtain reasonable assurance as to whether the statements made by the Company, including the summary of bias testing results presented in this report, are free from material misstatement, whether due to fraud or error.

Note that the criteria presented in this report were constructed specifically to address the requirements of a “bias audit” outlined in NYC Local Law No. 144 of 2021. The system was audited as though it were an automated employment decision tool (AEDT) under NYC Local Law No. 144 of 2021, but we do not make any determination whether the system is, in fact, an AEDT under this law.

Company Responsibilities

BABL AI Responsibilities

It is the responsibility of the lead auditor to express an opinion on the Company's assertions related to the bias testing of the system. In light of the current absence of generally accepted standards for the auditing of algorithms and autonomous systems, our examination was conducted in accordance with the standards and normative references outlined in this report.

Those standards require that we plan and perform audit procedures to obtain reasonable assurance about whether the assertions referred to above 1) satisfy the audit criteria and 2) are free of material misstatement, whether due to error or fraud. Within the scope of our engagement, we performed amongst others the following procedures:

• Inspection of submitted documents and external documentation

• Interviewing Company employees to gain an understanding of the process for

  determining the disparate impact and risk assessment results

• Observation of selected analytical procedures used in Company’s bias testing

• Inspection of the select samples of the bias testing data

• Inquiry of personnel responsible for governance and oversight of the bias testing and

  risk assessment

We believe that the procedures performed provide a reasonable basis for our opinion.

Independence

Our role as an independent auditor conforms to ForHumanity and Sarbanes-Oxley definitions of Independence. Fees associated with this contract are for the provision of the service to assess compliance. The payment of fees is unrelated to the decision rendered. Our decision is grounded solely in the criteria presented below.

Opinion

In our opinion, based on the procedures performed and the evidence received to obtain assurance, the bias testing and results presented by Company, as of 07/22/2024, is prepared, in all material respects, in accordance with the criteria outlined below.

Sincerely,

Shea Brown, Ph.D.
Lead Auditor, BABL AI Inc.

Emphasis of Matter

We emphasize several matters related to the dataset sourced by HackerRank to test for disparate impact: 1) self-declared demographic labels were not available for the system, so sex and race/ethnicity labels were inferred, 2) the use of inferred demographic labels means that the data is likely to be considered “Test Data” under § 5-3001, and 3) the amount and quality of the testing dataset (see Findings) were limited due the lack of historical data. Consequently, the conclusions drawn from the disparate impact quantification are subject to the limitations arising from the dataset and should be interpreted in light of this constraint. Our opinion is not modified with respect to this matter.

1 This is at the descresion of the NYC Department of Consumer and Worker Protection. 4

System Description

BABL AI was engaged to audit HackerRank’s testing of its Plagiarism Detection System.

From HackerRank: “Plagiarism detection system uses user-generated signals into an advanced machine-learning algorithm to flag suspicious behavior during an assessment. By understanding code iterations made by the candidate, the model can predict if they had major external help.

It’s important to note,... [that HackerRank] made the following efforts to prevent potential bias from PII or coding habits or question difficulty related bias:

• No race, gender or PII information in its training or at the time of inference.

• No typing speed data is used in the model since we consider typing speed is more like a

  personal habit instead of plagiarism indicators.

• Features are normalized in respect of the same question to minimize the bias from different question difficulties.

• Human-in-the-loop decision making: ML coding plagiarism [system] will never automatically drop candidates from the model prediction. Instead, we provide a comprehensive report of the suspicious attempt[s]... Our customers can review the detailed report and check the code replay by themselves to decide whether to move a candidate forward.”

Audit Summary

Background

New York City Local Law No. 144 of 2021 requires yearly “bias audits” for automated employment decision tools (AEDTs) used to substantially assist or replace decisions in hiring or promotion. Specifically, the law states that (1) the bias audit must “assess the [AEDTs’] disparate impact” on certain persons, (2) the audit must be conducted by an “independent auditor ... no more than one year prior to the use”, and (3) a “summary of the results of the most recent bias audit ... [must be] made publicly available on the website of the employer or employment agency.” The audit outlined in this document has been conducted to satisfy the law’s requirement for a bias audit only, and does not include other requirements such as candidate notifications. This report does not make any determination whether the system under this audit is, in fact, an automated employment decision tool as defined under NYC Local Law 144, or not.

Auditor Responsibilities

It is the responsibility of BABL AI auditors to:

1. Obtain reasonable assurance as to whether the statements made by the auditee are free from material misstatement, whether due to fraud or error,

2. Determine whether the statements made by the auditee provide sufficient evidence that the audit criteria (see Findings) have been satisfied, and

3. Issue an auditor’s report that includes an opinion.

As part of an audit in accordance with good auditing practice, BABL AI exercises professional judgment and maintains professional skepticism throughout the audit. Specifically, BABL AI auditors identify and assess the risks of material misstatement in documents provided by the auditee, perform audit procedures responsive to those risks, and obtain audit evidence that is sufficient and appropriate to provide a basis for our opinion, per Public Company Accounting Oversight Board (PCAOB)’s Auditing Standard 1105 on Audit Evidence,2 where applicable. In addition, this audit report follows International Standard on Assurance Engagements (ISAE) 3000’s guidelines on Assurance Report, where applicable.3

BABL AI is also responsible for maintaining auditors’ independence and objectivity to ensure the integrity of the opinion and certification provided. BABL AI as an organization, and all employee and contract auditors, adhere to strict independence as codified by the Sarbanes–Oxley Act of 20024 and the ForHumanity’s Code of Ethics.5 In addition, BABL AI Lead Auditors are ForHumanity Certified Auditors under NYC AEDT Bias Audit.6 For more details about our methodology and process, see Appendix – Audit Methodology.

2 https://pcaobus.org/oversight/standards/auditing-standards/details/AS1105
3 https://www.iaasb.org/publications/international-standard-assurance-engagements-isae-3000 -revised-assurance-engagements-other-audits-or-0

Scope & Objective

Out of Scope

1. The audit did not ensure sufficient testing of the tool’s disparate impact on any other protected class beyond race/ethnicity and gender

2. The audit did not certify that the system is “bias-free”

3. The audit is not intended for compliance purposes for any legislation other than the NYC Local Law No. 144

4 https://www.govinfo.gov/content/pkg/PLAW-107publ204/pdf/PLAW-107publ204.pdf
5 https://forhumanity.center/code-of-ethics/
6 https://forhumanity.center/nyc-bias-audit/

Conclusions

Our opinions for the bias audit of Plagiarism Detection System by HackerRank are as follows:

Findings

Note: The information disclosed under each criterion is not documentary evidence.

Disparate Impact Quantification

Components or combinations of components that were tested: N/A

Testing conducted by: HackerRank
Date of last testing: 05/10/2023
Time span of data: Nov 2022 – Feb 2023

Justification for the use of Test Data: From HackerRank: “Given the data protection contract with our customers, open source libraries are selected for name to race and name to gender estimation since no data transfer is required to a different vendor.”

PCVs for which disparate impact was quantified:

1. Gender
2. Race/ethnicity

PCVs for which disparate impact was not quantified:

1. Age

2. Immigration or citizenship status

3. Disability status

4. Marital status and partnership status

5. National origin

6. Pregnancy and lactation accommodations

7. Religion/creed

8. Sexual orientation

9. Veteran or Active Military Service Member status

Positive outcome: not being flagged by the system

User-configurable settings that can affect positive outcome: N/A

Settings on which disparate impact was tested: N/A

Method of quantifying disparate impact: selection rate defined as the rate at which candidates in a demographic group are not flagged by the system

Non-intersectional, Gender, sorted by Impact ratio

Non-intersectional, Race/ethnicity, sorted by Impact ratio

Intersectionals

Note: Data on these applicants was not included in the calculations above:

1. 123 applicants with an unknown gender category
2. 1053 applicants with an unknown race/ethnicity category, and
3. 1176 applicants with at least an unknown gender or an unknown race/ethnicity

7 N/A refers to the demographic group representing less than 2% of the total N applications in the table

Governance

Accountable party: Committee on Automated Decision Making Tools
Contact information: rohan.raman@hackerrank.com
Role in the auditee organization: Governance group/committee

Risk Assessment

Evidence of Risk Assessment completion: Risk Assessment Documents (risk assessment spreadsheet and narrative notes), and verbal testimony from Risk Assessment participants.

Appendix

Audit Methodology

The Process Audit

The BABL AI audit framework is the Process Audit, defined as “an impartial verification and evaluation process conducted by an independent auditor to determine whether sufficient evidence exists to warrant the judgment that AI risks have been prevented, detected, mitigated, or otherwise managed.” A process audit is modeled after the financial auditing practice as a cooperative third-party audit, and is distinguished from other commonly used forms of assessment of algorithms, such as first- or second-party assessments, and assurance.8 The audit framework contains three phases:

1. Scoping – The auditor conducts a preliminary survey of the auditee’s algorithm to gain a full understanding to contextualize documentary evidence

2. Evaluation & Verification – The auditee submits documentation containing evidence demonstrating satisfaction of the audit criteria which the auditors evaluate and verify.

3. Certification – If the auditee is determined to pass the audit criteria, the auditor drafts the auditor’s report and certifies the auditee’s algorithm.

Evaluation & Verification

The procedure for all BABL AI auditors in conducting a process audit follows the guidelines set forth in the Public Company Accounting Oversight Board (PCAOB)’s Auditing Standard 1105 on Audit Evidence, where applicable. Specifically, the auditors:

1. Obtain audit claims and statements from the auditee’s submitted documentation which either support or contradict the criteria and sub-criteria,

2. Evaluate the claims and statements in regard to satisfying the criteria and sub-criteria, based on the sufficiency and appropriateness of the evidence, and

3. Verify that the claims and statements made by the auditee are free from material misstatement, whether due to fraud or error.9

8 Carrier, R., & Brown, S. (2021). Taxonomy: AI Audit, Assurance & Assessment. ForHumanity.

https://forhumanity.center/blog/taxonomy-ai-audit-assurance-assessment/

9 “Reasonable assurance” is a high level of assurance but is not a guarantee that an audit conducted in accordance with good auditing practice always detects a material misstatement when it exists. Misstatements can arise from fraud or error and are considered material if, individually or in aggregate, they could reasonably be expected to influence the decisions of stakeholders taken based on these statements.

In addition, evaluation and verification of claims and statements may involve requesting additional supporting documentary evidence, and/or interviewing those responsible for the governance of the algorithm, other relevant employees of the auditee organization, or other third parties referenced in the submitted documentation.

At the end, the auditors reach an audit opinion based on:

1. The sufficiency and appropriateness of the audit evidence, and

2. The risk of material misstatement of the audit evidence.

10 Hasan, A., Brown, S., Davidovic, J., Lange, B., & Regan, M. (2022). Algorithmic Bias and Risk Assessments: Lessons from Practice. Digital Society, 1(1). https://doi.org/10.1007/s44206-022-00017-z