Accessing Data Stores from External Tools

While you could previously only access data stores using Lua DataStoreService in an experience server or Studio, you can now use Open Cloud DataStore API to access data stores from external scripts and tools. Open Cloud DataStore API is RESTful and provide almost the entire feature set as the Lua API.

You can also limit access to a subset of data stores needed for your tool when creating an API key, rather than exposing all data stores. This mitigates the impact in case your key gets leaked.

Usage

You can improve several areas of your workflow by accessing your data through Open Cloud DataStore API, including customer support, LiveOps, and data migration.

Customer Support Portal

Rather than using Studio or joining an experience every time, you can build a web application that allows your customer service agents to directly handle customer support requests, such as viewing and modifying player inventory and issuing refunds.

LiveOps Dashboard

You can code a big event in your experience while hiding it under a feature flag as part of the configuration data in a data store. You can then use your LiveOps dashboard to schedule the in-experience events by flipping the flag when it’s time for the event with DataStore API. The experience servers will detect this change by reading the flag and launching the event.

Data Migration

As your experience evolves, you might want to change your original data schema in a data store to accommodate the new features you want to launch. In order to not lose existing users’ data, you need to migrate your data store from the old schema to a new one. With DataStore API, you can write an external script that reads each entry from current data stores, maps the data to the new schema, and writes the entry back to a new data store.

Differences with the Lua API

Although the Open Cloud DataStore API is similar to the Lua DataStoreService, there are a few different requirements to be aware of:

  • Universe ID and data store name: The base URL for DataStore API is https://apis.roblox.com/datastores/v1/universes/{universeId}. Unlike Lua APIs, Open Cloud APIs are stateless and can come from anywhere, so you need to always provide the Universe ID, which is the identifier of your experience that you can get on the Creator Dashboard, and the data store name when sending the requests.

  • Separate permissions for creating and updating: The Lua API creates new entries if they don’t exist when you call GlobalDataStore:SetAsync(). The permissions for creating and updating entries using DataStore API are separate, which might be more desirable in certain situations. For example, a customer support tool should be able to edit a user’s profile, but not create a new one.

  • Data serialization: You need to serialize all data before network transportation. Serialization means to convert an object into that string, and deserialization is its inverse operation (convert string → object). The data store system uses JSON format. The Lua API serializes/deserializes entry content automatically. When using DataStore API, you need to generate or parse your entry data with JSON.

Security Permissions

Data stores usually store sensitive information, such as user profiles and virtual currency. To maintain security, each Open Cloud API has corresponding required permissions that you must add to your API key in order for the API call to work, such as the List Keys permission for the listing API. When you don’t add the required permissions, the API call returns an error. For the specific permissions that are required for each API, see DataStore API Reference.

When configuring your API keys, you can set granular permissions (read, write, list entry, etc.) for each data store within a specific experience, or you can give a key to read or write all data stores within an experience.

Getting the Universe ID

When sending requests to DataStore API, you need to pass the Universe ID, the identifier of the experience in which you want to access your data stores. You can obtain the Universe ID of an experience with the following steps:

  1. Navigate to the Creations page on the Creator Dashboard.
  2. Find the experience with data stores that you want to access.
  3. Click the ... button on the target experience's thumbnail to display a list of options, then select Copy Universe ID from the list.

Building Tools with DataStore API

You can use the language of your choice to build tools with DataStore API. This section provides a concrete example using Python to illustrate how you can list and read a subset of your users’ inventory, make edits, and then update back to an experience’s data store.

For this example, assume the following:

  • The name of the data store that stores player inventory is player_inventory.

  • The data schema for each data entry is "userId": {"currency": number, "weapon": string, "level": number}. The key is just userId.

  • The Python script lists a subset of user inventories based on prefixes, increase their virtual currency by 10 for a promotion, and update the data.

From the high level, you can build your Python app by adding API key permissions and adding the scripts.

Adding API key permissions

The example app requires three DataStore API methods to achieve its functionalities: List Entries, Get Entry, and Increment Entry, so you need to add the following API key permissions:

  • List keys
  • Read entries
  • Update entries

To create an API Key for this example:

  1. Go to the Credentials page on the Creator Dashboard.

  2. (Optional) If you want to create an API key for a group-owned experience, in the Creator section of the left-hand navigation, select the dropdown arrow, then select the group you would like to access the API key.

  3. On the upper-right of the screen, click the Create API Key button.

  4. Enter a unique name for your API key. Use a name that helps you recall its purpose later for continuous integration and deployment tooling access. This example uses the name Player-inventory-update.

  5. From the Select API System menu in the Access Permissions section, select DataStore API, then click the Add API System button.

  6. (Optional) In the DataStore API section, select API operations for specific data stores.

    1. Enable the Specific Data Store Operations toggle. By default, five data stores automatically load, but you can add additional data stores through the + Add Data Store to List button.
    2. Select the dropdown arrow next to a data store’s name, then select the API operations you want the data store to have access to.
  7. (Optional) Select API operations for the entire experience.

    1. Click the Select Experience to Add dropdown and select an experience.
    2. In the Experience Operations, click the dropdown arrow and select the operations you want to add to your API. This example selects Read Entry, Update Entry, and List Entry Keys for the entire experience.
  8. In the Security section, explicitly set IP access to the key using CIDR notation, and set an explicit expiration date so your key automatically stops working after that date. For this example, since you will do local testing first, you can remove the IP restriction by setting it to 0.0.0.0/0 and let it expire in 30 days.

  9. Click the Save and Generate key button.

  10. Copy and save the API key string to a secure location.

Adding the scripts

After creating the API Key with permissions required for the example app, you need to add Python scripts to perform app functionalities. The TutorialFunctions.py file shows how to define List Entries, Get Entry, and Increment Entry methods, and the Tutorial.py file uses the defined methods to list a subset of user inventories, increase the virtual currency for each user, and update the data. The scripts use API_KEY to signify this API key.

TutorialFunctions.py

1import hashlib
2import requests
3import json
4import base64
5
6class DataStores:
7 def __init__(self):
8 self._base_url = "https://apis.roblox.com/datastores/v1/universes/{universeId}"
9 # API Key is saved in an environment variable
10 self._apiKey = str(os.environ['API_KEY'])
11 self._universeId = "UNIVERSE_ID"
12 self.ATTR_HDR = 'Roblox-entry-Attributes'
13 self.USER_ID_HDR = 'Roblox-entry-UserIds'
14 self._objects_url = self._base_url +self._universeId+'/standard-datastores/datastore/entries/entry'
15 self._increment_url = self._objects_url + '/increment'
16 self._version_url = self._objects_url + '/versions/version'
17 self._list_objects_url = self._base_url +self._universeId+'/standard-datastores/datastore/entries'
18
19 def _H(self):
20 return { 'x-api-key' : self._apiKey }
21 def _get_url(self, path_format: str):
22 return f"{self._config['base_url']}/{path_format.format(self._config['universe_id'])}"
23
24 return r, attributes, user_ids
25
26def get_entry(self, datastore, object_key, scope = None):
27 self._objects_url = self._base_url +self._universeId+'/standard-datastores/datastore/entries/entry'
28 headers = { 'x-api-key' : self._apiKey }
29 params={"datastoreName" : datastore, "entryKey" : object_key}
30 if scope:
31 params["scope"] = scope
32 r = requests.get(self._objects_url, headers=headers, params=params)
33 if 'Content-MD5' in r.headers:
34 expected_checksum = r.headers['Content-MD5']
35 checksum = base64.b64encode(hashlib.md5(r.content).digest())
36 #print(f'Expected {expected_checksum}, got {checksum}')
37
38 attributes = None
39 if self.ATTR_HDR in r.headers:
40 attributes = json.loads(r.headers[self.ATTR_HDR])
41 user_ids = []
42 if self.USER_ID_HDR in r.headers:
43 user_ids = json.loads(r.headers[self.USER_ID_HDR])
44
45 return r
46
47 def list_entries(self, datastore, scope = None, prefix="", limit=100, allScopes = False, exclusive_start_key=None):
48 self._objects_url = self._base_url +self._universeId+'/standard-datastores/datastore/entries'
49 headers = { 'x-api-key' : self._apiKey }
50 r = requests.get(self._objects_url, headers=headers, params={"datastoreName" : datastore, "scope" : scope, "allScopes" : allScopes, "prefix" : prefix, "limit" : 100, "cursor" : exclusive_start_key})
51 return r
52
53 def increment_entry(self, datastore, object_key, incrementBy, scope = None, attributes=None, user_ids=None):
54 self._objects_url = self._base_url +self._universeId+'/standard-datastores/datastore/entries/entry/increment'
55 headers = { 'x-api-key' : self._apiKey, 'Content-Type': 'application/octet-stream' }
56 params={"datastoreName" : datastore, "entryKey" : object_key, "incrementBy" : incrementBy}
57 if scope:
58 params["scope"] = scope
59
60 r = requests.post(self._objects_url, headers=headers, params=params)
61 attributes = None
62 if self.ATTR_HDR in r.headers:
63 attributes = json.loads(r.headers[self.ATTR_HDR])
64 user_ids = []
65 if self.USER_ID_HDR in r.headers:
66 user_ids = json.loads(r.headers[self.USER_ID_HDR])
67
68 return r
69
Tutorial.py

1import tutorialFunctions
2
3DatastoresApi = tutorialFunctions.DataStores()
4
5# Set up
6datastoreName = "PlayerInventory"
7
8# List keys for a subset of users (you might need to use the nextPageCursor to view other entries)
9keys = DatastoresApi.list_entries(datastoreName)
10print(keys.content)
11
12# Read inventory for each user
13for x in range(5):
14 updatedObjectKey = "User_"+str(x+1)
15 value = DatastoresApi.get_entry(datastoreName, updatedObjectKey)
16 # change response type to a string
17 updatedValue = value.json()
18 print(updatedObjectKey + " has "+str(updatedValue)+" gems in their inventory")
19# Update the currency of each user by 10
20for x in range(5):
21 updatedObjectKey = "User_"+str(x+1)
22 value = DatastoresApi.increment_entry(datastoreName, updatedObjectKey, 10)
23 # change response type to a string
24 updatedValue = value.json()
25 print(updatedObjectKey + " now has "+str(updatedValue)+" robux in their inventory")
26

To test, set the API_KEY environment variable and run Tutorial.py file:


1export API_KEY=... \
2python Tutorial.py
3

Legacy Scope Support

This section is only relevant if you use the legacy Scope feature.

Like Lua DataStoreService, every key in a data store has a default global scope, but you can further organize keys by setting a unique string as a scope that specifies a subfolder for the entry. Once you set a scope, it automatically prepends to all keys in all operations done on the data store.

The scope categorizes your data with a string and a separator with "/", such as:

Key Scope
houses/User_1 houses
pets/User_1 pets
inventory/User_1 inventory

All data store entry operation methods have a Scope parameter for when you need to access the entries stored under a non-default scope. For example, you might have a 1234 key under the default global scope, and the same key under special scope. You can access the former without using the scope parameter, but to access the latter, you have to specify the scope parameter as special in Get Entry or Increment Entry API calls.

Additionally, if you want to enumerate all the keys stored in a data store that has one or multiple non-default scopes, you can set the AllScopes parameter in List Entries method to be true, in which case the call returns a tuple with key string and scope. In the previous example, the List Entries would return both (1234, global), and (1234, special) in the response.

You cannot pass Scope and AllScopes parameters on the same request, otherwise the call returns an error. Leveraging the helping functions from DataStore API module, the following code illustrates how you can read every key in a data store with a custom scope:


1# Set up
2import tutorialFunctions
3DatastoresApi = tutorialFunctions.DataStores()
4datastoreName = "PlayerInventory"
5
6# List keys for global scope
7specialScopeKeys = DatastoresApi.list_entries(datastoreName, scope = "global", allScopes = False)
8print(keys.content)
9# List keys for special scope
10specialScopeKeys = DatastoresApi.list_entries(datastoreName, scope = "special", allScopes = False)
11print(keys.content)
12# List keys for allScope set to true
13specialScopeKeys = DatastoresApi.list_entries(datastoreName, allScopes = True)
14print(specialScopeKeys.content)
15

Response for global scope:


1{ "keys": [{ "scope": "global", "key": "User_2" }], "nextPageCursor": "" }
2

Response for special scope:


1{"keys":[{"scope":"special","key":"User_6"},{"scope":"special","key":"User_7"}],"nextPageCursor":""}'
2

Response for AllScopes = True:


1"{\"keys\":[{\"scope\":\"global\",\"key\":\"User_3\"},{\"scope\":\"global\",\"key\":\"User_4\"},{\"scope\":\"global\",\"key\":\"User_5\"},{\"scope\":\"special\",\"key\":\"User_6\"},{\"scope\":\"special\",\"key\":\"User_7\"}],\"nextPageCursor\":\"\"}"
2