Experiments

Experiments let you run in-game and matchmaking A/B tests to measure the causal impact of changes to your game. For example, you can show different onboarding experiences to different players and measure the difference in playtime, retention, and other key performance indicators.

Experiments are excellent for measuring the following:

• Engagement - Onboarding flows, progression systems, control schemes, custom matchmaking
• Monetization - Shop visibility and user experience, starter pack types, pricing

Create experiments

Experiments come in two types:

• In-game experiments let you measure the impact of different config values.
• Matchmaking experiments let you measure the impact of different custom matchmaking configurations. Unlike in-game experiments, you can only run one matchmaking experiment at a time.

1. If you don't already have a config, create one for your game.

2. On the Creator Hub Experiments page for your game, click Create experiment.

3. For Type, choose In-experience.

4. Specify a name, goal metric, and planned duration for the experiment. Experiments run for between 14-60 days.

   Regardless of what you choose as your goal metric, experiments track all metrics in the list.

5. Choose a percent rollout. This number is the percentage of players that you want to include in the experiment.

   In general, the more people you include in an experiment, the better the data, but use your judgment on what's best for your game.

   Pay attention to the minimum detectable effect, which is the smallest change in the selected metric that the experiment can reliably detect.

6. Specify variants and percentages.

   Variants are alternative values for your config. For a numeric config key bossHealth with a control value of 500, you might specify a variant of 300. You can have up to two variants and one control in an experiment.

   Percentages dictate how to assign variants within the experiment rollout. Consider the following example:

   • You choose an overall rollout of 40%.
   • You specify two variants and a 50/50 split between them and the control.

   In this example, 60% of your users are excluded from the experiment; these users receive the control and have no impact on experiment results. Approximately 20% of your users receive the control as part of the experiment. Another 20% receive the variant. Depending on your player count, this distribution might not be large enough to yield actionable results.

   When in doubt, we recommend simple 50/50 experiments of one variant and one control on large rollouts. The experiments are easier to configure, and the results are easier to interpret.

   

7. (Optional) Target the experiment to a specific audience so that only matching players are eligible for enrollment. Targeting uses the same player attributes as conditional configs, and you can copy existing conditions directly from your configs. For more information, see Target experiments to specific audiences.

   

8. The final step is scheduling. You can start experiments immediately or schedule them for a later date and time. After you schedule an experiment, you can't change its configuration (duration, rollout percentage, variants, etc.), but you can reschedule it.

Metrics

Experiments track all of the following metrics over the experiment duration. These metrics are updated every 24 hours while the experiment is running. However, during the first 24 hours of your experiment, Playtime, Payer conversion rate, and ARPU also update every 5 minutes so you can catch critical regressions (early harm) quickly.

Experiment status

The Experiments page shows the following statuses for experiments.

Target experiments to specific audiences

By default, an in-game experiment can enroll any player in your rollout percentage. Targeting lets you run the experiment on a specific audience instead, such as players in certain countries, specific tenure windows, or active spender tiers. Targeting uses the same attributes as config targeting.

Set up targeted experiments

You configure targeting criteria when you create the experiment. After the experiment starts, you can't change the targeting rules, the control value, the variants, or any conditional rules that the config key uses.

You can reuse existing conditional rules from your configs to define your targeting, but the experiment stores an independent copy of each rule. Editing or deleting the original config condition later has no effect on the experiment's targeting.

Because narrowing your audience reduces your sample size, Roblox updates the minimum detectable effect (MDE) using an estimate of the audience that matches your targeting criteria. Make sure your targeted audience is large enough to produce statistically meaningful results.

How Roblox resolves targeted values

Roblox evaluates experiments first, before any rules from standard conditional configs. When a config key has an active experiment, Roblox resolves the value in this order:

1. The active experiment value, if the player is targeted and enrolled.
2. The first matching config condition.
3. Any subsequent matching config conditions.
4. The default value.

Players in the control group receive the value that your existing config rules produce. You can't run more than one active experiment on the same config key at a time.

Per-session evaluation

Roblox evaluates targeting attributes per session, so an individual player's eligibility can change over time:

• If a player matches your targeting criteria during a session, they can be enrolled and receive an experiment variant.
• If they no longer match in a later session, they stop receiving the experiment value and immediately fall back to your standard config rules. If your experiment needs continuity beyond eligibility—for example, a multi-session onboarding flow targeted at new players—persist the value yourself, such as in a data store.

A player's data stays attributed to the variant (or control) they were enrolled in, even if they later stop matching the criteria. For example, D7 retention still counts a player who enrolled on day 0 and whether they returned on day 7, even if they no longer qualify by day 4.

Config changes while an experiment runs

To prevent configuration drift and corrupted experiment data, Roblox locks the config key the moment an experiment starts running:

• You can't edit or delete the running config key, nor can you modify its standard conditional values.
• Any global conditions that the active config references are also locked. You can't edit, delete, or reorder them relative to each other.

The lock releases when the experiment completes.

Add experiments to your code

Applying in-game experiments is similar to applying configs. The main difference is the use of ConfigService:GetConfigForPlayerAsync() rather than ConfigService:GetConfigAsync().

GetConfigForPlayerAsync() retrieves a player-specific snapshot. When you call GetValue(), the snapshot checks for an active experiment and enrolls (or doesn't enroll) the user based on the rollout percentage.

local ConfigService = game:GetService("ConfigService")
local Players = game:GetService("Players")

local function onPlayerAdded(player)
    local playerConfig = ConfigService:GetConfigForPlayerAsync(player)
    local leaderboardColor = playerConfig:GetValue("leaderboardColor")
end

Players.PlayerAdded:Connect(onPlayerAdded)

• You must call GetConfigForPlayerAsync() separately for each player; GetConfigAsync() does not apply experiments.

• After you call GetValue() on a player-specific snapshot, the player associated with the snapshot is enrolled in the experiment for that key and that key only. All subsequent calls to the method return the same control or variant for the duration of the experiment. Only the first call is random.

  Wait to call GetValue() until you need it. Calling GetValue() too early can cause you to enroll players who never interact with the part of the game you're experimenting on.

• Enrollment in experiments isn't limited to new users. Even if a user previously received a value from GetConfigAsync(), you can still enroll them in an experiment using a player-specific snapshot from GetConfigForPlayerAsync().

• If a key in a player-specific snapshot doesn't have an active experiment, GetValue() returns the standard config value (or nil if it has no value).

Custom enrollment

If you want to enroll only players that meet criteria based on in-game state, you have to write additional code to check for those criteria and only then call GetValue() to enroll them in the experiment. Consider the following example:

• You want to test a new control scheme in your racing game.
• The scheme is for advanced players, so you want to target players who have won a large number of races.

Your code might look something like this:

local function getControlScheme(player, racesWon)
    if racesWon < 20 then
        return "standardScheme"
    else
        -- Player has many wins, enroll in experiment
        local playerConfigSnapshot = ConfigService:GetConfigForPlayerAsync(player)
        if playerConfigSnapshot:GetValue("useNewControlScheme") then
            return "newScheme"
        else
            return "standardScheme"
        end
    end
end

If you want the control scheme to persist on subsequent sessions, you likely need to add a value to the player's entry in a data store.

View and interpret results

Click View to see details and results. In the Details And Progress tab you can see the total number of players enrolled, as well as the number of players that received the control value and each variant. Viewing this page early in the experiment is useful strictly for making sure the experiment is running properly, not for taking action. Before taking action, see Best practices.

Early harm metrics

For the first 24 hours, or until the first daily results land, early harm metrics update every 5 minutes so you can catch unintended critical harm early. Click Metrics to see these results. Look for critically harming metrics highlighted in red.

Hover over a metric and click View confidence to see the confidence interval.

During this window, results test only for harm, so the confidence interval has no lower bound, only an upper bound. A metric is flagged as critically harmful only if the entire confidence interval falls below the critical harm threshold. In the following example, Playtime is at -12.67% with an upper bound of -2.73%. Even though the estimate is lower than the critical harm threshold of -10% and the entire confidence interval is below 0%, the metric is not classified as critically harmful because the interval is not entirely below the harm threshold.

Use these results only to guide early stopping decisions in the first 24 hours. Do not use them for launch decisions, rely on daily results instead.

Early harm thresholds

Each metric has its own early harm threshold. If a variant's lift falls below this threshold, it will be classified as critically harmful to that metric. However, for metric variant comparisons with fewer than 10,000 players enrolled across the variant and the control, variant lifts are shown but decisions on harm will not be made until the sample size provides enough data to reliably detect harm.

Early harm notifications

If critical harm is detected in your experiment during the early harm analysis period, a notification is delivered to the game owner and experiment owner for early action. Notifications are delivered via email, Creator Hub notification tray, and through an optional webhook. Upon receiving an early harm notification, you should review the experiment metrics and decide whether or not the experiment should be stopped early.

Daily results

After an experiment has run for at least 24 hours, click Results to see the latest results, which update every 24 hours. Look for statistically significant changes in goal metrics, which the dashboard highlights in green or red. These changes are more likely to show the impact of your variant and less likely to be false positives or negatives.

A metric is statistically significant when the confidence interval for its percent change does not overlap with 0%. In the following example, ARPU is up 2.37%, with lower and upper bounds 0.55% and 4.19%, which makes the change statistically significant.

For convenience, the results page lets you replace the default config value with one of the variants from the experiment.

Sample ratio mismatch

If your experiment is failing to enroll users into the variants in the expected proportions, an alert banner for Sample Ratio Mismatch (SRM) will appear in the experiment results tab. If SRM is detected, it is recommended to stop and restart your experiment.

SRM is checked every 5 minutes for the first 24 hours of your experiment, and then daily after. SRM can invalidate experiment results; do not make launch decisions based on compromised and unreliable results.

Make a decision

When your experiment concludes, click Make decision to start a guided rollout. Roblox shows any warnings about statistical significance and experiment duration, then prompts you to select a winning variant or keep the control. Based on your choice, Roblox rolls out the appropriate change to your permanent configs; returning to the Configs page shows the new value. Click Change winner if you change your mind.

If you have unresolved staged changes on the config when you complete the experiment, Roblox temporarily stashes them and restores them on a best-effort basis after the rollout.

Your decision determines the config changes that Roblox proposes:

• Control: Roblox makes no changes to your existing configs.

• Untargeted variant: The variant value becomes the config's new default value. Roblox removes all other conditions on that config key, though they remain in your global conditions list.

• Targeted variant: Roblox integrates the winning rule back into your permanent config using the first applicable option in this priority order:

  1. Overwrite: If an identical condition already sits at the top of the config, Roblox overwrites its value with the winning variant's value.
  2. Add existing: If a matching global condition exists and would naturally sit at the top, Roblox adds it to the config with the variant's value.
  3. Promote: If a matching global condition can be safely promoted to the lowest rank needed to make it the top rule in this config (without altering other configs), Roblox adds it to the config and reorders it globally.
  4. Create new: Otherwise, Roblox creates a new global condition ranked one level above the highest condition currently used in the config. If this creates a duplicate rule inside the config, Roblox strips the redundant conditional value while preserving the global rule.

Best practices for experiments

• Use the minimum detectable effect (MDE) to decide if your experiment is worth running.

  Roblox calculates MDE using your goal metric and number of players per variant, which is based on daily active users, rollout percentage, experiment duration, and variant splits. If the MDE is too high for your goal metric (for example, more than 100%), it's unlikely you can reach statistical significance. Games with fewer than 1,000 daily active users might struggle to get useful data from experiments.

  

• Start with a hypothesis. Rather than just changing a variable and checking the results, write a cause-and-effect statement about what you changed, what you expect to happen, and why. As you experiment more and more, having a set of written hypotheses to accompany your results can help clarify your thinking and spark new ideas for experiments.

• Let experiments run for their full durations. The novelty effect (temporary interest in a change not because it's better, but because it's new) can heavily skew early results, sometimes causing them to swing in and out of statistical significance. Ending experiments early increases the odds of you taking premature action based on anomalous spikes that more data would have smoothed out or even contradicted.

  • The exception is early harm in the first 24 hours: if metrics are flagged as critically harming or you see sample ratio mismatch, review and consider stopping. Do not use early harm results for launch decisions.

• Don't act without statistical significance. Even seemingly large changes in player behavior might not be statistically significant, generally due to small sample size. If a change isn't statistically significant, ignore it.

• Avoid changes during experiments. Major bugs of course need fixes, but changes to game content can impact player behavior and invalidate your results, even if the changes seem unrelated to your experiment. Similarly, only run experiments simultaneously if you're confident they won't interact with each other.

• Use confidence intervals for deep dives into metrics and to check for borderline cases of statistical significance. If the confidence interval is too wide, the metric might never reach statistical significance.

• If one metric is significantly up and another significantly down, you have to decide whether the trade-off is worth it, possibly in conjunction with other statistically significant movements.

• Experiments provide strong signal, but statistical significance deals in probabilities, not certainties—hence the confidence interval. Data variability, sample size, and magnitude of the change all impact the probability of detecting whether a variant affected player behavior. Any action you take based on the results of an experiment should be balanced against qualitative data like player feedback and your overall vision for the game.

• Document your findings and decisions. Even if you don't use them to run additional experiments, having a body of knowledge and evidence can inform how you design your games.