
What Is a Test? Meaning, Types and How A/B Tests Work

A test is a structured way to find something out: what a person knows, whether a product works, or whether an idea is true. In marketing and conversion rate optimization (CRO), a test is an experiment on a live website: you show real visitors different versions of a page and measure which one gets more people to take the action you care about.
This guide covers both meanings. First the plain definition, then the version marketers use every day: A/B tests, A/A tests, the metrics that matter and the mistakes that quietly ruin results.
TL;DR
- A test is a procedure for examining something to find out whether a thing is present, true or working correctly.
- In digital marketing, a test (usually an A/B test) splits live traffic between a control and a variant and compares them on a predefined goal.
- An A/A test shows the same page to both groups. It checks that your testing setup is trustworthy before you rely on it.
- Pick one primary metric and a few guardrail metrics before you launch, change one thing at a time, and do not stop early because the numbers look good.
- You can start for free: Stellar's Sandbox plan costs $0 per month.
Table of Contents
- What does "test" mean?
- What a test means in digital marketing
- Common types of tests marketers run
- What is an A/A test?
- How to run a test, step by step
- What to measure
- Mistakes that make a test meaningless
- Good first tests to try
- Our perspective on testing
- Start testing without a developer
- FAQ
- Sources
What does "test" mean?
The everyday meaning is simple. The Cambridge Dictionary describes a test as a way of asking questions or setting practical activities to measure knowledge or ability. A school exam and a driving test both fit that definition.
The same word works for things, not just people. Longman defines a test in the product sense as a process to discover whether equipment or a product works correctly, or to learn more about it. A crash test on a car and a stress test on software belong here.
Science and statistics use a stricter version. The APA Dictionary of Psychology defines a test as a procedure used to examine or determine whether a factor or phenomenon is present, and also uses the word for a set of statistical operations that judge whether a hypothesis is valid. That second sense is the one hiding behind every marketing experiment.
The common thread across all of these: you set up a situation, observe what happens, and use the result to decide something. If the outcome cannot change your decision, it was not really a test.
What a test means in digital marketing
In CRO, the test is almost always an A/B test. The Nielsen Norman Group describes it as a quantitative research method that compares two or more design variations in a live product with live users, using predetermined business-success metrics to find the best performer.
The moving parts are few:
- Control (A): the current experience, left as it is.
- Variant (B): the change you want to try.
- Traffic split: real visitors are divided so each person sees only one version. NN/g notes the versions should ideally differ in one element.
- Goal: the action that decides the winner, such as a signup, a click or a purchase.
Other publishers frame it the same way. HubSpot calls a CRO test an experiment designed to test strategies intended to maximize conversion rate, and Shopify describes CRO testing as running website experiments to see which changes lead more visitors to take a desired action, such as signing up or buying.
A researcher-level description is the same idea stripped down: an A/B test randomly assigns users to a control and a modified treatment, then computes and compares key metrics. Random assignment is what lets you say the change caused the difference, rather than the season, the ad campaign or the device mix.
If you want the broader picture of why this discipline pays off, this overview of conversion rate optimization tips for digital marketers shows where testing fits among the other levers.
Common types of tests marketers run
The word "test" covers several formats. Knowing which one you are running keeps you from reading the wrong conclusion into the data.
| Test type | What you compare | Best used for |
|---|---|---|
| A/B test | Original vs. one changed version | Headlines, buttons, images, form length, offers |
| A/B/n test | Original vs. several variants | Trying a few distinct ideas at once, when traffic allows |
| Multivariate test | Combinations of several elements | Finding how elements interact, on high-traffic pages |
| Split URL test | Two separate pages | Full redesigns or very different layouts |
| A/A test | Identical versions | Checking your tool and setup for errors |
For most small and mid-sized sites, the plain A/B test is the right default. Multivariate tests need far more traffic because every combination needs enough visitors to be measured, and few small sites have that.
What is an A/A test?
An A/A test shows the same page to both groups. Nothing is different, so there should be no real winner. The test exists to check the experiment itself, not the page.
The logic is straightforward. If your setup is working, the two identical groups should convert at about the same rate and the tool should report no significant difference. The NYU-hosted paper on online experiments notes that rigorous A/A tests can validate that an experimentation system correctly finds no significant difference.
Here is a worked example with made-up numbers. You split 20,000 visitors evenly between two identical pages. Version A converts 1,003 of 10,000 visitors and version B converts 1,007. That 0.04 point gap is noise and the tool should call it inconclusive. If instead it declared a confident winner, something is wrong with how visitors are assigned, how conversions are counted or how the statistics are computed.

Why run an A/A test?
There are three practical reasons:
- Catch implementation errors. The same research describes how A/A tests can expose invalid experiments or incorrectly applied formulas. A misfiring goal, a duplicated tag or a broken split shows up here before it contaminates a real test.
- Trust your numbers. The broader point from that paper is that trustworthy experimentation means validating the measurement system itself, because attractive results can otherwise come from implementation or analysis mistakes.
- Learn your baseline. Running identical versions shows how much your conversion rate naturally wobbles, which helps you judge how big a real lift needs to be before it is believable.
An A/A test is most useful when you adopt a new tool, change your analytics setup or start testing on a new type of page. It is not something you need to repeat before every experiment.
What an A/A test cannot tell you
It cannot tell you whether any particular idea will work, and a single clean A/A result does not prove every future test is valid. Even correct systems will flag a false difference now and then by chance, so a one-off significant result on identical pages is a reason to investigate, not automatically proof of a broken tool.
How to run a test, step by step
The process is the same whether you use a spreadsheet and a developer or a visual editor.
- Start with a problem, not a design. Look at where visitors drop off. A pricing page with high traffic and low signups is a better target than a low-traffic about page.
- Write a hypothesis. One sentence: "If we change X to Y, then metric Z will improve because of reason R." The reason keeps you learning even when the variant loses.
- Choose one change. Mailchimp's explanation of the scientific method behind A/B tests makes the case that a valid comparison changes one thing while keeping everything else controlled. Change the headline and the image and the button together, and you will not know which one mattered.
- Define the goal and metrics first. More on this below.
- Split traffic randomly so each visitor sees one version and keeps seeing it on return visits.
- Let it run. Decide the sample size or minimum duration before launch and do not stop early because one line is ahead.
- Analyze, then record what you learned whether the variant won, lost or tied.
Mailchimp also stresses that a valid test needs a defined campaign, test element and goal. If you cannot fill in those three blanks, you are not ready to launch.
What to measure
The goal you pick decides what "winning" means, so choose it deliberately. NN/g lists conversion rate, click-through rate, bounce rate, retention rate and revenue per user as common test metrics.
It also recommends separating metrics into two roles:
- Primary metric: the behavior you intend to change, such as completed signups.
- Guardrail metrics: checks that the change did not hurt the wider business, such as revenue per user or retention.
This matters more than it looks. A pop-up that lifts email signups can quietly reduce purchases. If you only watch the local metric, you declare victory while the business loses. NN/g's guidance is to define primary and guardrail metrics before interpreting results, so you cannot pick the flattering number after the fact.
Mistakes that make a test meaningless
- Peeking and stopping early. Early gaps often shrink. Set your stopping rule in advance.
- Testing too many things at once. You get a result but no explanation.
- Skipping the hypothesis. Without it, a win teaches you nothing you can reuse.
- Ignoring segments you cannot trust. A 5,000-visitor test split across ten segments leaves each one too small to read.
- Running tests with a slow script. A testing tool that delays your page can change visitor behavior and flicker the original content before the variant loads, which muddies the very thing you are measuring.
- Trusting an unverified setup. If goals fire twice or traffic is not split evenly, every result is suspect. This is the problem an A/A test is built to catch.
For a deeper look at reading results once the data is in, see our testing analysis guide for marketers.
Good first tests to try
If you have never run one, start where the effect is easy to see and the change is easy to make:
- Headline copy on your main landing page. Our write-up on how to optimize landing page headline copy walks through angles worth trying.
- Video vs. image in the hero area, covered in this video versus image test idea.
- Call-to-action wording and placement, one element at a time.
- Form length, removing a single field and watching completions.
For more starting points, browse our collection of A/B testing ideas, or read the complete guide to A/B testing for landing pages for a full workflow.
Our perspective on testing
Most teams do not fail at testing because they lack ideas. They fail because testing feels heavy: a developer ticket, a slow script, a dashboard nobody trusts. Each of those frictions quietly reduces how many experiments get run, and the number of experiments you run is the best predictor of how much you learn.
That is why we built Stellar around speed and simplicity. Our script is designed to be lightweight, at 5.4KB, so the tool itself has minimal impact on the page you are trying to improve. A no-code visual editor means a marketer can set up a variant without waiting on engineering. Goal tracking and real-time analytics are there so you can define your primary metric up front and watch it without exporting data.
We also believe small and mid-sized teams deserve the same disciplined approach as large ones: one change at a time, a hypothesis written down, guardrails chosen before launch. The tool should make that discipline easy, not optional. Run the A/A test when you set up, then spend your energy on the tests that can actually move revenue.
Juan
Start testing without a developer
If you are ready to move from definitions to your first live experiment, Stellar is built for exactly this. It offers a no-code visual editor, dynamic keyword insertion for personalized landing pages, advanced goal tracking and real-time analytics, with integrations for WordPress, Shopify, Webflow, Wix, Squarespace, Framer and Bubble.

Pricing is based on monthly tracked users. As published on our site, Sandbox is $0 per month, Plus is $29 per month, Pro is $99 per month and Scale is $399 per month. Enterprise has no published price. You can see the details and start with a plan on the Stellar site, and if you want a quick conversion tactic to pair with your first test, try the free FOMO popup generator.
FAQ
What do you mean by a test?
A test is a procedure for finding something out, such as what someone knows, whether a product works or whether a hypothesis holds. In marketing, it means an experiment that compares versions of a page, email or ad on a defined goal.
What is the difference between an A/B test and an A/A test?
An A/B test compares two different versions to see which performs better. An A/A test compares two identical versions to confirm your tool and setup report no difference when none exists. Run A/A to validate the system, then A/B to improve the page.
How long should a test run?
Long enough to reach the sample size you calculated before launch, and across full weekly cycles so weekday and weekend behavior are both included. Ending the moment one version pulls ahead is a common way to get a false winner.
How many things should I change in one test?
One. NN/g advises that versions ideally differ in a single element, which lets you attribute any change in results to that element. If you want to test several changes together, treat it as a separate multivariate test with far more traffic.
Do I need a developer to run a test?
Not necessarily. A no-code visual editor lets a marketer edit headlines, images and buttons directly and launch the experiment. Complex changes to logic or back-end behavior may still need engineering support.
Sources
- TEST | definition in the Cambridge English Dictionary
- test | meaning of test in Longman Dictionary
- APA Dictionary of Psychology: test
- A/B Testing 101, Nielsen Norman Group
- What is a CRO Test?, HubSpot
- What Is CRO Testing and How To Conduct It, Shopify
- A/B Testing in Digital Marketing, Mailchimp
- The Surprising Power of Online Experiments