Data management reference

Data schema

For the full schema of the readings table (document format, column reference, the data column, and per-component data structures), see Captured data schema.

Query reference

Indexed fields and query optimization

You can improve query performance by filtering on indexed fields early in your query. Viam stores data in blob storage using the path pattern:

/organization_id/location_id/robot_id/part_id/component_type/component_name/method_name/capture_day/*

The more specific you can be, starting with the beginning of the path, the faster your query. These fields are indexed:

• organization_id
• location_id
• robot_id
• part_id
• component_type
• component_name
• method_name
• capture_day

Additional optimization techniques:

• Filter and reduce data early. Use $match (MQL) or WHERE (SQL) before expensive operations like grouping or sorting.
• Use $project early to drop unneeded fields from the processing pipeline.
• Use $limit or LIMIT while developing queries to avoid scanning your entire dataset.
• For frequent queries on recent data, use the hot data store.
• For recurring queries (dashboards), use data pipelines to pre-compute materialized views.

Supported MQL operators

Viam supports a subset of MongoDB aggregation pipeline stages. Operators not on this list will return an error.

• $addFields
• $bucket
• $bucketAuto
• $count
• $densify
• $fill
• $geoNear
• $group
• $limit
• $match
• $project
• $redact
• $replaceRoot
• $replaceWith
• $sample
• $set
• $setWindowFields
• $skip
• $sort
• $sortByCount
• $unset
• $unwind

See the MQL documentation for syntax details.

SQL limitations

Viam supports the MongoDB Atlas SQL dialect:

• If a database, table, or column identifier begins with a digit, a reserved character, or conflicts with a reserved SQL keyword, surround it with backticks (`) or double quotes (").
• To include a single quote in a string literal, use two single quotes (use o''clock to represent o'clock).
• The date data type is not supported. Use timestamp instead.

For a full list of limitations, see the MongoDB Atlas SQL Interface Language Reference.

Date queries

MQL time-range queries perform better with the BSON date type than with $toDate. Use JavaScript Date() constructors in mongosh:

use sensorData

const startTime = new Date('2024-02-10T19:45:07.000Z')
const endTime = new Date()

db.readings.aggregate([
    { $match: {
        time_received: {
            $gte: startTime,
            $lte: endTime
        }
    }}
])

Permissions

Users with owner or operator roles at the organization, location, or machine level can query data. See Role-Based Access Control for details.

Supported resources

The following components and services support data capture and cloud sync. The table shows which capture methods are available for each resource type. Not all models support all methods listed for their type.

If the resource type you need is not listed, you can still capture data from it using the DoCommand method with a custom docommand_input parameter.

Capture and sync configuration

This section describes the configuration fields for data capture and cloud sync.

Data management service attributes

The data management service controls sync behavior, storage paths, and deletion policies. Most of these settings are configured through the Viam app UI. Edit JSON directly for settings not exposed in the UI, such as deletion thresholds, sync thread limits, and MongoDB capture.

{
  "services": [
    {
      "name": "my-data-manager",
      "api": "rdk:service:data_manager",
      "model": "rdk:builtin:builtin",
      "attributes": {
        "sync_interval_mins": 1,
        "capture_dir": "",
        "tags": [],
        "capture_disabled": false,
        "sync_disabled": true,
        "delete_every_nth_when_disk_full": 5,
        "maximum_num_sync_threads": 250
      }
    }
  ]
}

{
  "services": [
    {
      "name": "my-data-manager",
      "api": "rdk:service:data_manager",
      "model": "rdk:builtin:builtin",
      "attributes": {
        "capture_dir": "",
        "tags": [],
        "additional_sync_paths": [],
        "sync_interval_mins": 3
      }
    }
  ]
}

Platform-managed service settings

The following settings appear in your machine’s configuration but are not processed by viam-server on your machine. They are read and enforced by the Viam cloud platform:

Data capture method attributes

Data capture is configured per-resource in the service_configs array of a component or service. When you configure capture through the Viam app UI, these fields are set automatically. The table below is the JSON-level reference for manual configuration.

Here is where capture attributes live in a component’s JSON config:

{
  "name": "my-sensor",
  "api": "rdk:component:sensor",
  "model": "rdk:builtin:fake",
  "service_configs": [
    {
      "type": "data_manager",
      "attributes": {
        "capture_methods": [
          {
            "method": "Readings",
            "capture_frequency_hz": 0.2,
            "additional_params": {},
            "disabled": false
          }
        ]
      }
    }
  ]
}

Platform-managed capture settings

The following settings are processed by the Viam cloud platform, not by viam-server. retention_policy is set at the attributes level (sibling to capture_methods), while recent_data_store is set inside an individual capture_methods[] entry:

{
  "type": "data_manager",
  "attributes": {
    "capture_methods": [
      {
        "method": "Readings",
        "capture_frequency_hz": 0.2,
        "recent_data_store": {
          "stored_hours": 24
        }
      }
    ],
    "retention_policy": {
      "days": 30
    }
  }
}

For remote parts capture, see Capture from multi-part machines.

Local storage

This section describes how captured data is stored on the machine before syncing.

Capture directories

By default, captured data is stored in ~/.viam/capture. The actual path depends on your platform:

2025-01-15T14:27:26.073Z    INFO    rdk    server/entrypoint.go:77    Starting viam-server with following environment variables    {"HOME":"/home/johnsmith"}

You can change the capture directory with the capture_dir attribute in the data management service attributes.

Automatic deletion

After data syncs successfully, it is automatically deleted from local storage. While a machine is offline, captured data accumulates locally.

Automatic deletion triggers when all of these conditions are met:

• Data capture is enabled
• Local disk usage is at or above the disk_usage_deletion_threshold (default: 90%)
• The capture directory accounts for at least the capture_dir_deletion_threshold proportion of disk usage (default: 50%)

Control deletion behavior with the delete_every_nth_when_disk_full attribute.

Micro-RDK

The micro-RDK (for ESP32 and similar microcontrollers) supports data capture with a smaller set of resources than viam-server. See the Micro-RDK tab in the supported resources table for the specific methods available.

On micro-RDK devices, captured data is stored in the ESP32’s flash memory until it is uploaded to the cloud. If the machine restarts before all data is synced, unsynced data since the last sync point is lost.