Frequently Asked Questions

Question 1

How should clients allocate their campaign budgets between creative production, media and creative pre-testing of variants?

Answer

Ultimately, while you should strive to push as much budget as possible into media to solve for frequency and reach, that spend is only effective if underpinned by strong creative concepts. Our clients typically spend 5% of their media budget on creative pre-testing. Creative pre-testing serves as essential insurance to ensure the message resonates.

Question 2

What is the most effective methodology for testing early-stage creative concepts?

Answer

Early stage testing is a crucial part of the creative development process. Before developing a suite of ads across all media platforms and formats, early-stage testing provides solid foundations for effective messaging - we identify the most compelling scenes, help clarify messaging, uncover emotional responses, and ensure your brand or product stands out in the story.

We can test your creative concept and ideation at various stages of development. Early-stage concepts are tested in the form of boardomatics or animatics. We also produce Gen-AI video concepts, which provide a more user-friendly and realistic draft and flow of the final video shoot.

For production-ready concepts, our HMM Pro solution evaluates effectiveness of the creative concepts in a simulated media environment to show how your creative performs in real-world conditions. Watch this explainer video for more details.

Question 3

How many concepts should I test at once?

Answer

It is helpful to test multiple concepts to maximize your chance of success, as in practice about half of creatives struggle to have an incremental impact on brand outcomes. Even in the absence of brand lift, we will be able to score which one will have better resonance and outcomes based on supporting predictive diagnostics. We always recommend testing a variety of lengths, both short form and long form ads so they can be adapted across your entire media plan effectively.

Question 4

When you are using humans in the testing process, do you use a general population sample?

Answer

We tailor the audience to the campaign target audience in our survey testing. We work with all the largest global panels (companies which source survey respondents) as well as specialised local market panels, focusing on high data quality provision and audience representation in our screening processes. We are panel-agnostic in each market to maximize feasibility with both broad and niche audiences, and profile survey respondents to ensure a match with your desired audience.

Question 5

How does HMM achieve higher predictive accuracy than traditional testing vendors?

Answer

An important part of our program with clients is our Trust But Verify initiative. This is a pro-active comparison of creative testing results with true campaign brand lift results (for example brand lift studies conducted directly with the live campaign on YouTube). Through this feedback loop over many years, we have seen that several factors are important in driving the predictive accuracy of creative testing.

alignment of the audience to the true target audience of the campaign.
alignment of the media platform and format.
localization, market and vertical level insights and benchmarks are not transferable.

Other solutions which test ads against a general population; compare to general population norms; or use forced-exposure TV style ads to understand campaigns that will run in social media - will all struggle to uncover a true creative performance signal.

Finally we also encourage our clients not to forget the other important component of campaign success - high quality, well targeted and executed media strategies are also needed.

Question 6

How does HMM’s methodology quantify and validate creative risk?

Answer

HMM quantifies creative risk by identifying two specific areas of failure: Growth Risk and Reputational Risk. Our methodology validates these risks before media launch to prevent campaign failure and brand erosion:

‍Growth Risk (Predicting Performance Failure): We measure the probability of an ad failing to achieve incremental brand lift. This represents the inability for an ad to drive growth. ‍
Reputational Risk (Safeguarding Brand Health): We surface audience concerns on issues such as controversial talent/influencer selection, culturally insensitive messaging or perceived stereotypes/discrimination. This impacts reputational risk - where well intentioned messages can accidentally lean into audience fears and anxieties.

Question 7

Can HMM test content for fragmented media environments (TikTok, Reels, Programmatic)?

Answer

Yes, we have the ability to test against an unrivalled number of media platforms and formats across 30 markets. We can test either with HMM Pro for high-investment media or with HMM AI for all your long-tail activity.

Question 8

How does Human Made Machine differ from firms like Nielsen or Kantar?

Answer

Similarly to these organisations we have a global footprint and world-class researchers.

In contrast our foundations are very different.

Our broader team consists of data scientists, engineers, and both media and creative strategy leads. This means we understand your end-to-end creative development and optimisation process and can integrate seamlessly with your organisation. When working with us, your end-to-end project or program will be managed and overseen by a senior researcher, offering consultation at every stage of the process.

As an organisation our singular focus is creative effectiveness, making us true specialists in creative strategy and optimisation. We actively research new creative trends, provide unbiased recommendations of creative optimisation and best practices.

Finally, it’s all in our name. As a modern organisation, with founders with backgrounds in data science, data engineering, and full-stack development, it is our machine that powers the answers to your key questions on creative quality and effectiveness. We are consistently recognised for our ability to integrate seamlessly with client and media systems, our provision of high-quality well structured data, and extremely high levels of automations.

Question 9

Why is HMM more robust than "fast" automated testing platforms?

Answer

Our speed is highly competitive but we ultimately prioritise rich, human-centred insights and focus on quality. We allow a little extra time for briefing to make sure the project setup perfectly answers the key questions you need answering. We combine over 40 automated data quality checks with human verification. You can find more about our data quality checks here.

Question 10

Can HMM support global campaigns across multiple languages and cultures?

Answer

Human Made Machine supports multiple large scale global testing programs for brands like Google, Coach and the NBA. As a team we speak over 13 languages and have 5 regional hubs that support over 30 markets.

All of our testing environments are fully localized (language, content) and we integrate regional trends, local survey-design best practices enabling us to consistently answer two universally applicable questions - what ads are working and why?

Question 11

How does HMM protect the security and intellectual property of pre-launch creative assets?

Answer

Depending on the sensitivity of the material we use standard or enhanced security to protect leaking of your ad data.

Standard Security

Double NDAs: All respondents agree to binding confidentiality terms from both the panel and Human Made Machine.
Single-Use Links: Unique access prevents respondents from re-entering or sharing the survey.
Obscured Assets: Test materials use non-indexed, obscure URLs to prevent discovery.
Lifecycle Management: Links and assets are deactivated immediately after fieldwork concludes.
Referrer Tracking: Access is logged and restricted to authorized domain referrers.
Traceability: Respondents are uniquely identified; any breach results in tracing and a permanent ban from future research.

Enhanced Security

Dynamic Watermarking: For high-sensitivity projects, unique identifiers are overlaid on assets to deter and track unauthorized screen captures or sharing.

Question 12

What makes HMM the preferred choice for Tier 1 global brands like Google?

Answer

HMM has been through rigorous competitive screening processes and tested head to head with other major creative testing providers to make the cut. Our clients have primarily cited the ability of HMM creative testing to predict real world outcomes and drive returns as their primary motivation for working with us. In addition our strategic recommendations and the high quality of our data have also been referenced.

Question 13

How much budget is typically 'wasted' on unoptimized creative?

Answer

It’s common for as little as 50% of creative to drive an incremental impact and can be even more challenging in high clutter/low attention environments. If you’re not testing creative, the success of your media budget becomes at best a coin-toss.

Question 14

What's the average time to run a test, from recruitment to the initial insights readout?

Answer

For the HMM Pro test results (includes both what works and why) it takes 1 week for 2-4 creative for a gen pop audience for an initial readout. For HMM AI (primarily validation of what works), allow for 24-48 hours depending on the volume of ads.

Question 15

Can HMM AI test all our creatives, or do we also need to use Pro?

Answer

HMM AI can be used to test all creatives in your media plan, except for OOH (Out-of-Home). However, a high accuracy prediction with AI is enabled by us customising our model with data specific to your brand. As such we recommend a complementary testing program of both HMM Pro and HMM AI. Test high investment hero assets and lead concepts with Pro. This will provide the most accurate read on incremental brand lift, give you an in depth understanding on the ‘why’ and support better AI-prediction scores.

Question 16

We do Creative Pre-Testing today, but our provider shares a "long-term brand" resonance score. Is that not important?

Answer

At HMM we also provide emotional resonance scores as an important additional diagnostic measure. However, on its own, resonance can be highly disconnected from brand outcomes. For example ads may be emotionally resonant but the audience may not associate the ad at all to the brand. As a result, in isolation these measures can be a poor predictive signal of incremental impact and difficult to validate against in market results. By focusing on Brand Lift, we are always measuring the incremental impact, aligning our recommendation specifically to brand growth.

Question 17

How do you measure brand marketing campaigns that are targeting brand health metrics?

Answer

It’s important to understand brand incrementality on multiple levels - how does exposure to the ad creative support incremental brand lift, how does each channel drive incremental brand lift, and how does the campaign as a whole drive lift.

Measure creative impact through creative pre-testing with HMM Pro or HMM AI testing for your true target audience and media.
Measure the channel level impact through Brand Lift Studies (BLS). These are available across all major digital channels for free.
Larger companies spending a lot of money should use campaign lift studies, measuring brand lift across the entire marketing spend. We do not specialise in such studies.

For brand marketing campaigns, methodologies which support "true test and control" methodologies (randomized control trials) are the most reliable. Focus on the "big three KPIs": unaided brand awareness, consideration, and purchase intent. We recommend avoiding metrics like "ad recall" which have a poor relationship with revenue outcomes.

Question 18

What is the ROI of brand lift on long-term revenue?

Answer

Brand lift drives medium and long-term revenue. Some general guidelines from our CSO:

If solving for purchase intent, you will see revenue impact within three to six months.
If solving for consideration, you will see an impact on revenue within six to nine months.
If solving for un-aided brand awareness, you will see the lift over nine to fifteen months.

The actual amount of revenue impact varies by company. We recommend reviewing the whitepaper “Profit Ability 2: The new business case for advertising report” which assesses the long term impact of brand on revenue by vertical for 141 brands.

Question 19

Have you tested the accuracy of the model by testing the creatives marked as failures?

Answer

Yes, testing some failed creative is an important part of verifying the predictive accuracy of creative testing - we just recommend this testing is limited in scope due to the risk to your media budget. Creative testing is also not a binary measure in practice and you will have strong, average and weak assets which will be reflected in campaign performance and can be captured in the scoring. At HMM, “Trust but Verify”, a process to understand and improve predictive accuracy, is an important part of our program with large brand advertisers.

Question 20

If a creative performs well on one channel but poorly on another, how do you recommend validating and adapting creatives across platforms?

Answer

It is normal for a creative to perform differently across channels, such as TV versus YouTube, or YouTube versus YouTube Shorts. It is important to test across channels and adapt to the user experience in that channel.

For example, HMM Pro tests how an ad will work in a skippable ad environment versus non-skippable, or how it works in your social feeds.

Our recommendation is to use HMM Pro across every channel where there is a large budget being spent, and use HMM AI to test high-volume creatives (e.g., variants across Reels, TikTok, Shorts).

For multinational companies, it is important to test the concepts with HMM Pro across different geographies to understand the "what and the why" across every culture. An ad that resonates in the US may not work in Japan. Once you understand the 'what and why' across cultures with HMM Pro, you can then rapidly test high-volume executions with HMM AI.

HMM also regularly shares best practices for different ad formats on our LinkedIn page.

Question 21

Which media channel provides the most reliable environment for creative pre-testing?

Answer

One of the biggest common errors we see is that advertisers will focus on long form ads, and adapt them to short form. In practice an ad that works in long form is much harder to adapt to short form. If your media plan is reliant on short form content, start there. If your creative development process is staggered, concept testing can be a powerful way of understanding message resonance. In this process we look at the most impactful brand cues and memorable scenes that can be used to create shorter form content.

Question 22

What is HMM AI?

Answer

HMM AI is a proprietary AI model designed to predict brand lift and rapidly remove low-potential creative from executions and long-tail media activity.This pure AI assessment provides a high-speed methodology for creative validation at scale. It acts as a primary filter for creative executions, allowing brands to optimize high-volume, long-tail media assets with scientific accuracy before they go live.

Question 23

What data is the HMM AI model trained on?

Answer

HMM’s proprietary AI model is trained on a decade of high-sample, in-context HMM Pro test results and ad metadata. The dataset includes over 10,000 creatives across 200+ brands and more than 30 markets. Our extensive history of running high-quality in-lab data from large-sample randomized control tests ensures strong foundational accuracy for predicting brand lift and core diagnostic measures of creative effectiveness.

Question 24

Does HMM AI take into account specific audience targeting for its predictions?

Answer

Yes, HMM AI produces different scores for different audience segments by treating the audience as a key input layer in its neural network. The model captures highly structured audience data, including demographics (age/gender), platform (iOS vs. Android), category purchase intent, and B2B segments. This allows the model to predict how the same creative will perform across various tailored audience groups.

Question 25

How does HMM ensure the integrity and accuracy of its AI predictions?

Answer

HMM maintains model integrity by ensuring that all training flows from a consistent source of independent HMM Pro brand lift experiment data to the AI model - never feeding the model with its own outcomes. This "neutral learning process" prevents feedback loops that can reinforce bias. Additionally, we use real-world in-market performance as an input to refine the model. In combination these measures ensure the model remains adaptive rather than stagnant.

Question 26

What are the limitations of using AI for creative testing?

Answer

While AI excels at driving consistency and speed, it can have a backward-looking perspective that makes it less equipped to adapt to novel, spontaneous trends like TikTok phenomena or new brand launches. Additionally, AI may struggle with complex human behaviors or superficial emotional understanding, sometimes lacking the intuitive cultural nuance that humans possess when interpreting playful or unique scenes. HMM Pro and HMM AI have a symbiotic relationship, the more brand lift experiments you run with HMM Pro, the stronger the ability of HMM AI to predict unique elements of your brand creative.

Question 27

Does HMM AI use my creative or data to train the model for other clients?

Answer

No, HMM AI does not use client creative or data to train the model for other clients. We operate with a strict "neutral learning process" where the training data only flows from independent HMM Pro in-lab results to the AI model. This ensures that your pre-launch assets and proprietary data remain secure and are never used to inform the competitive landscape or reinforce biases in predictions for other brands. We create custom models for clients which can incorporate their own HMM Pro data into their AI brand lift prediction results.

Question 28

How does HMM AI interpret human behavior or creative scenes?

Answer

HMM AI may struggle with complex human behaviors or superficial emotional understanding as it can lack the intuitive cultural nuance that humans possess. While the model is highly effective at driving consistency and speed, it can sometimes find it difficult to interpret playful or unique scenes. This is why we recommend AI for rapid, high-volume filtering, while human-led Pro testing remains the gold standard for high-stakes brand storytelling and unique concepts.

Question 29

How is the HMM AI model updated and refined over time?

Answer

The HMM AI model is refined using real-world in-market performance as a feedback loop to ensure the model remains adaptive rather than stagnant. By constantly updating the model with fresh performance data and metadata from over 200 brands, we maintain a high-fidelity predictive engine. This continuous learning process ensures that our foundational accuracy for performance lift stays aligned with the evolving media landscape.

Question 30

How does HMM AI account for regional differences in creative effectiveness?

Answer

HMM AI provides regionally aligned predictions by utilizing a dataset that spans more than 30 global markets. Our methodology rejects the "global average" fallacy, accounting for the significant variance in breakthrough thresholds and cultural friction points across different geographies. Because the bar for effectiveness depends on the context, HMM AI applies market-specific benchmarks to every assessment, ensuring assets intended for APAC or LATAM markets are not penalized or rewarded based on US or European performance data.