BigQuery streaming export

Stream mapped Nimriz events into a customer-owned BigQuery table with a Google Cloud service account.

Availability: BigQuery setup is available on Enterprise plans.

Requirements

• An Enterprise plan.
• A Google Cloud project, dataset, and existing table.
• A service-account JSON key with permission to update table data and read the target table and dataset.

Granting BigQuery Data Editor on the destination dataset covers the normal write path. Keep the service account limited to the intended dataset. BigQuery documents the exact permissions for tabledata.insertAll.

Create the default table

Replace the project, dataset, and table names before running this example.

CREATE TABLE `PROJECT_ID.DATASET_ID.TABLE_ID` (
  account_id STRING NOT NULL,
  organization_id STRING,
  event_type STRING NOT NULL,
  event_id STRING NOT NULL,
  timestamp TIMESTAMP NOT NULL,
  url_id STRING,
  domain_id STRING,
  domain_name STRING,
  short_code STRING,
  short_url STRING,
  destination_url_capped STRING,
  touch_type STRING,
  country STRING,
  region STRING,
  city STRING,
  device_category STRING,
  browser_family STRING,
  os_family STRING,
  referrer_host STRING,
  social_source STRING,
  ip_hash STRING,
  utm_source STRING,
  utm_medium STRING,
  utm_campaign STRING,
  utm_term STRING,
  utm_content STRING,
  variant_key STRING,
  conversion_id STRING,
  conversion_type STRING,
  conversion_value FLOAT64,
  conversion_currency STRING,
  gclid STRING,
  wbraid STRING,
  gbraid STRING
);

If you customize Nimriz property mapping, update the BigQuery schema to match the resulting keys and types. external_id is never included in warehouse payloads.

Connect BigQuery in Nimriz

1. Create a service-account JSON key for the scoped service account.
2. Go to Integrations›Destinations›Browse catalog and choose BigQuery.
3. Paste the Service account JSON and enter the Project ID, Dataset ID, and Table ID.
4. On Review, confirm that the test will leave a sample row, then select Save, test, and enable.
5. Nimriz encrypts the key, saves the destination disabled, performs a real streaming insert, and enables delivery only when that test succeeds.

If the provider test fails, the saved destination stays disabled. Correct the key, table, or schema, then select Retry test and enable.

Changing the service account or target table invalidates the prior test.

Delivery behavior

Nimriz sends batches of up to 100 mapped rows through tabledata.insertAll. Each row uses its delivery ID as BigQuery's insertId. Nimriz inspects indexed insert errors even when BigQuery returns HTTP 200, so valid rows can succeed independently of rejected rows.

Streaming inserts are subject to the quotas of the project and dataset location that own the destination table. Review Google's current BigQuery streaming insert quotas and monitor quotaExceeded responses when sizing sustained delivery.

Troubleshooting

• Permission denied: verify the service account has access to the target dataset and table.
• Row schema error: compare the mapped payload keys and types with the table schema.
• Not found: confirm project, dataset, and table IDs without display-name substitutions.
• Rotate the key: replace the JSON key in Nimriz, test it, then disable the prior key in Google Cloud.