Experiments

Experiments let you run in-game and matchmaking A/B tests to measure the causal impact of changes to your game. For example, you can show different onboarding experiences to different players and measure the difference in playtime, retention, and other key performance indicators.

Experiments are excellent for measuring the following:

• Engagement - Onboarding flows, progression systems, control schemes, custom matchmaking
• Monetization - Shop visibility and user experience, starter pack types, pricing

Create experiments

Experiments come in two types:

• In-game experiments let you measure the impact of different config values.
• Matchmaking experiments let you measure the impact of different custom matchmaking configurations. Unlike in-game experiments, you can only run one matchmaking experiment at a time.

1. If you don't already have a config, create one for your game.

2. On the Creator Hub Experiments page for your game, click Create experiment.

3. For Type, choose In-experience.

4. Specify a name, goal metric, and planned duration for the experiment. Experiments run for between 14-60 days.

   Regardless of what you choose as your goal metric, experiments track all metrics in the list.

5. Choose a percent rollout. This number is the percentage of players that you want to include in the experiment.

   In general, the more people you include in an experiment, the better the data, but use your judgment on what's best for your game.

   Pay attention to the minimum detectable effect, which is the smallest change in the selected metric that the experiment can reliably detect.

6. Specify variants and percentages.

   Variants are alternative values for your config. For a numeric config key bossHealth with a control value of 500, you might specify a variant of 300. You can have up to two variants and one control in an experiment.

   Percentages dictate how to assign variants within the experiment rollout. Consider the following example:

   • You choose an overall rollout of 40%.
   • You specify two variants and a 50/50 split between them and the control.

   In this example, 60% of your users are excluded from the experiment; these users receive the control and have no impact on experiment results. Approximately 20% of your users receive the control as part of the experiment. Another 20% receive the variant. Depending on your player count, this distribution might not be large enough to yield actionable results.

   When in doubt, we recommend simple 50/50 experiments of one variant and one control on large rollouts. The experiments are easier to configure, and the results are easier to interpret.

   

7. The final step is scheduling. You can start experiments immediately or schedule them for a later date and time. After you schedule an experiment, you can't change its configuration (duration, rollout percentage, variants, etc.), but you can reschedule it.

Metrics

Experiments track all of the following metrics over the experiment duration.

Experiment status

The Experiments page shows the following statuses for experiments.

Add experiments to your code

Applying in-game experiments is similar to applying configs. The main difference is the use of ConfigService:GetConfigForPlayerAsync() rather than ConfigService:GetConfigAsync().

GetConfigForPlayerAsync() retrieves a player-specific snapshot. When you call GetValue(), the snapshot checks for an active experiment and enrolls (or doesn't enroll) the user based on the rollout percentage.

local ConfigService = game:GetService("ConfigService")
local Players = game:GetService("Players")

local function onPlayerAdded(player)
    local playerConfig = ConfigService:GetConfigForPlayerAsync(player)
    local leaderboardColor = playerConfig:GetValue("leaderboardColor")
end

Players.PlayerAdded:Connect(onPlayerAdded)

• You must call GetConfigForPlayerAsync() separately for each player; GetConfigAsync() does not apply experiments.

• After you call GetValue() on a player-specific snapshot, the player associated with the snapshot is enrolled in the experiment for that key and that key only. All subsequent calls to the method return the same control or variant for the duration of the experiment. Only the first call is random.

  Wait to call GetValue() until you need it. Calling GetValue() too early can cause you to enroll players who never interact with the part of the game you're experimenting on.

• Enrollment in experiments isn't limited to new users. Even if a user previously received a value from GetConfigAsync(), you can still enroll them in an experiement using a player-specific snapshot from GetConfigForPlayerAsync().

• If a key in a player-specific snapshot doesn't have an active experiment, GetValue() returns the standard config value (or nil if it has no value).

Targeted enrollment

If you want to target some portion of your players that meet specific criteria, you have to write additional code to check for those criteria and only then call GetValue() to enroll them in the experiment. Consider the following example:

• You want to test a new control scheme in your game.
• You don't want to include existing players (who are presumably used to the existing scheme), only new players.

Your code might look something like this:

local function getControlScheme(player, racesCompleted)
    if racesCompleted > 0 then
        return "standardScheme"
    else
        -- Player is new, enroll in experiment
        local playerConfigSnapshot = ConfigService:GetConfigForPlayerAsync(player)
        if playerConfigSnapshot:GetValue("useNewControlScheme") then
            return "newScheme"
        else
            return "standardScheme"
        end
    end
end

If you want the control scheme to persist on subsequent sessions, you likely need to add a value to the player's entry in a data store.

View and interpret results

After an experiment has run for at least 24 hours, click View to see details and results.

You can see the total number of players enrolled, as well as the number of players that received the control value and each variant. Viewing this page early in the experiment is useful strictly for making sure the experiment is running properly, not for taking action. Before taking action, see Best practices.

After the experiment is complete, check the Results tab. Look for statistically significant changes in goal metrics, which the dashboard highlights in green or red. These changes are more likely to show the impact of your variant and less likely to be false positives or negatives.

Hover over any metric to see the View confidence button, which shows the confidence interval.

A metric is statistically significant when the confidence interval for its percent change does not overlap with 0%. In the following example, D1 retention is up 17.4%, with lower and upper bounds 8.02% and 22.03%, which makes the change statistically significant.

For convenience, the results page lets you replace the default config value with one of the variants from the experiment. Click Make decision to choose a variant or Change winner if you change your mind. If you then return to the Configs page, you should see the new value.

Best practices for experiments

• Use the minimum detectable effect (MDE) to decide if your experiment is worth running.

  Roblox calculates MDE using your goal metric and number of players per variant, which is based on daily active users, rollout percentage, experiment duration, and variant splits. If the MDE is too high for your goal metric (for example, more than 100%), it's unlikely you can reach statistical significance. Games with fewer than 1,000 daily active users might struggle to get useful data from experiments.

  

• Start with a hypothesis. Rather than just changing a variable and checking the results, write a cause-and-effect statement about what you changed, what you expect to happen, and why. As you experiment more and more, having a set of written hypotheses to accompany your results can help clarify your thinking and spark new ideas for experiments.

• Let experiments run for their full durations. The novelty effect (temporary interest in a change not because it's better, but because it's new) can heavily skew early results, sometimes causing them to swing in and out of statistical significance. Ending experiments early increases the odds of you taking premature action based on anomalous spikes that more data would have smoothed out or even contradicted.

• Don't act without statistical significance. Even seemingly large changes in player behavior might not be statistically significant, generally due to small sample size. If a change isn't statistically significant, ignore it.

• Avoid changes during experiments. Major bugs of course need fixes, but changes to experiegamence content can impact player behavior and invalidate your results, even if the changes seem unrelated to your experiment. Similarly, only run experiments simultaneously if you're confident they won't interact with each other.

• Use confidence intervals for deep dives into metrics and to check for borderline cases of statistical significance. If the confidence interval is too wide, the metric might never reach statistical significance.

• If one metric is significantly up and another significantly down, you have to decide whether the trade-off is worth it, possibly in conjunction with other statistically significant movements.

• Experiments provide strong signal, but statistical significance deals in probabilities, not certainties—hence the confidence interval. Data variability, sample size, and magnitude of the change all impact the probability of detecting whether a variant affected player behavior. Any action you take based on the results of an experiment should be balanced against qualitative data like player feedback and your overall vision for the game.

• Document your findings and decisions. Even if you don't use them to run additional experiments, having a body of knowledge and evidence can inform how you design your games.