Data integrity in the AI era

At Human Made Machine (HMM), we help you understand how effective your ads are with our creative pre-testing solutions. We define “good AI” as very human - our HMM AI predictions are driven by millions of real people who have seen ads in our media environments and participated in our brand surveys.

But keeping insights human-centered is a moving target. As AI becomes more accessible, its use among survey-takers has also increased - contributing to a rise in survey fraud. Promisingly, survey fraud - often overlooked by legal systems - is now being targeted by the U.S. Department of Justice. But from bots and click farms to AI-generated responses and low-quality human input, the threat of poor data quality is growing. For those of us who rely on true human input, it’s essential to understand how these risks can affect data quality - and what can be done to eliminate them.

Our commitment to human-centered, high-quality insights

In today’s blog, we focus on our commitment to maintaining high-quality, human-centered creative insights. That’s why having our robust quality control measures in place - to prevent, detect, and remove AI-generated and low-quality human responses - is essential.

HMM sampling quality starts at the source

Poor-quality partners lead to poor-quality data, so finding the best one matters. We work with the world’s largest global sample providers, as well as partners with specialist market coverage. They are the foundation of credible, high-quality insights.

We source and assess the reliability of these partners’ sample through screening and testing:

• Where does their sample come from?
  
• Are they effective at reaching well-profiled audiences at scale?
  
• Are they transparent and knowledgeable about their sampling processes?
  
• How do they verify that their survey-takers are real humans, not AI bots?
  

While no provider is entirely fraud-free, a trustworthy partner should clearly explain their quality standards and control methods. These early conversations help us assess how dependable they’ll be when we identify fraud, and how strong their existing processes are.

Still, real-world performance can vary. That’s why we test every partner for audience representation and run our own 20-step quality check to maximize accuracy.

We identify audiences that are more vulnerable to fraud

At HMM, we assess the specific fraud and quality risks linked to different target audiences. Different groups show different fraud patterns. For example:

• B2B audiences: Often face higher fraud rates due to larger pay incentives.
  
• Younger audiences: Are more likely to take surveys on mobile, especially young males - who also tend to submit lower-quality open-ended answers.
  
• Device type: Desktop users are more likely to paste AI-generated responses and show click-farm behavior. Mobile users, meanwhile, are more prone to disengagement and low attention.
  

Understanding these nuances allows us to identify vulnerabilities early and embed the right checks into our surveys before we start recruiting the audience.

Our surveys are weaponized to catch fraud

Survey questions can do more than collect insights - they can expose low-quality data.

We design our surveys to detect inconsistencies and inattention:

• Contradictory answers: For example, saying an ad is both “enjoyable” and “irritating” may signal a bot or inattention.
  
• Open-ended questions: These are powerful for identifying quality. We look for relevance, logic, and natural language. AI-generated responses often contain long word counts, list formatting, or unusual punctuation like em dashes (—).
  
• Answer consistency: If someone selects the same answer across a series of statements, that’s a quality flag.
  
• Trap questions: We use and regularly refresh AI-resistant trap questions to ensure ongoing effectiveness.
  

HMM security measures are the backbone of fraud prevention

Strong back-end security is essential in survey research. We enforce the following controls:

• Preventing duplicate or repeat entries.
• Excluding survey-takers based on prior participation.
• Blocking navigation back to previous questions.
• Requiring all questions to be answered before proceeding.
• Use of a CAPTCHA at the start to block bots.
• Disabling copy/paste in the survey environment is another effective measure - particularly for AI bots.
• Collecting metadata (IP address, time zone, device type) to confirm that survey-takers are located within the target market.
• Recording timestamps throughout the survey to identify patterns of fraud such as speedsters, laggers or spikes in activity.*

There is no single check that will remove all fraud, but each one adds a layer of protection. We have tested the efficacy of each of these checks and our data shows, that in combination, they dramatically reduce AI-generated and poor quality survey responses.

^{*Speeders: Those who rush through without reading. Laggers: Those who pause for too long, possibly to look up answers. Spikes: Large spikes in completions during overnight hours, or sharp drops in participation often signal fraud.}

Our proprietary fraud algorithm is backed by millions of survey responses

Even the best sample provider will have inconsistencies. Not every survey-taker provides thoughtful, high-quality data. And relying on any single flag in isolation can cause over-cleaning, driving up costs and reducing true audience coverage.

After over 10 years of surveying millions of people, we’ve developed a proprietary algorithm that evaluates these quality signals simultaneously. Our system:

• Targets the automated removal of at least 95% of poor-quality data
• Minimizes the loss of valid, high-quality data (less than 5% of high-quality data)
  

Every Step Matters

From knowing your audience to securing your systems and processes, protecting data quality is a full-spectrum effort.

Through our commitment to combating fraud, HMM deliver human-centered creative insights that our clients can trust.

‍

‍Want to learn more about how to maximize the return on your ad spend with effective creative? Connect with us and request a demo.