Some of these metrics are crucial for building disqualifying logic in the Alchemer surveys.
| Category | Metric | Definition | Recommended Action |
|---|---|---|---|
|
Search
| |||
| Search | search flag | Identifies previous access to the survey; 1 if true and 0 if false. | Recommendation: term if ‘1’ if you are not allowing return access to the survey |
| Search | search_threat_ potential_score | Threat potential scores range from 1 (lowest threat) to 100 (highest threat). | Default disqualification in template: term if search_ threat_potential_score >= 29. We recommend NOT adjusting this metric. |
| Search |
search_threat_ potential
|
Scored "low", "medium", or "high" Informed by the threat_potential_score parameter. | Ignore: redundant to above |
| Search | search_failure_ reason |
This parameter will populate with a failure reason if one from the list below occurs:
|
|
|
Activity
token1 = represents the Research Defender digital fingerprint of the respondent token2 = represents the IP address of the respondent | |||
| Activity | activity_count_ token1 | The number of occurrences of the Research Defender digital fingerprint found across the industry within the past 24hrs. |
Recommendation: After data collection, consider cleaning out the top 6% of respondents with the highest activity_count (RepData conducted a 3rd party analysis of Research Defender – found that cleaning out the top 6% of activity_count_token2 respondents alone increased data quality)
|
|
Activity
|
activity_count_ token2 | The number of occurrences of the IP Address found across the industry (tracked by Research Defender’s systems) within the past 24hrs. |
Recommendation: After data collection, consider cleaning out the top 6% of respondents with the highest activity_count
|
|
Activity
|
activity_average_token1 | The average number of occurrences of all Research Defender digital fingerprints found across the industry (tracked by Research Defender’s systems) within the past 24hrs. | Do not use to disqualify – helpful to use as context when comparing with each respondent’s activity_count_token1 value |
|
Activity
|
activity_average_token2 | The average number of occurrences of all IP Addresses found across the industry within the past 24hrs. |
Do not use to disqualify – helpful to use as context when comparing with each respondent’s activity_count_token2 value
|
|
Review
| |||
| Review | review_flag | Triggers as 1 if language is unknown, profanity is detected, or copy/pasted response is detected. | Recommendation: term based on "flag" if the presence of at least one of the three scenarios is disqualification-worthy. |
| Review | review_language_detected |
Text description of the language used in the response. |
Default disqualification in template: term if review_language_detected = Unknown. We recommend NOT adjusting this metric.
|
| Review | review_language_detected_score |
Confidence interval with which the language is correctly detected: 0.1 = most confident |
Recommendation: disqualify if review_language_detected = 1 (adds some extra backup strength to review_language_detected metric above, but not necessary) |
| Review | review_ similarity_text |
Triggers as 1 if there was a previous survey response to the same open-end that exactly matches the current response – helps to detect bots or scripts that provide identical responses. Will only run the "review_similarity_text" check if the response is 20 characters or longer. |
Recommendation: disqualify if "similarity_text" = 1 |
| Review | review_profanity_check |
Detects if profanity is used in response; 1 if true and 0 if false. Review_profanity_check_score records the same – redundant metric. |
Recommendation: disqualify if "review_profanity_check" = 1 |
| Review | review_pasted_ response | Detects if any part of the text response was pasted from the user’s clipboard; 1 if true and 0 if false. | Recommendation: If you want to disqualify if any part of the response is pasted, disqualify if "review_pasted_response" = 1 |
| Review | review_pasted_ response_score | The value represents the % of the number of characters pasted from the clipboard in the answer. For example, if 40% of the characters in the open-end were pasted from the respondent's clipboard, review_pasted_response_score will be 0.4. | Default disqualification in template: term if review_pasted_response_score >= 0.5. Adjust as needed/desired. |
| Review | review_ composite_score | This score uses all other flags to produce a quality score, from 0 to 100. A lower score indicates a response with fewer flags. A higher score indicates a flawed answer for reasons ranging from profanity to copy/paste etc. Responses with a language equal to ‘Unknown’ and responses that trigger the Pasted Response, Profanity, and Similarity flags will be set to 100. |
We do not recommend cleaning data based on review_composite_score. Use as a general health metric and disqualify based on more specific criteria.
|