Auto-generated from OPERATOR_REGISTRY
Total operators: 61
This is the definitive reference for all built-in transform operators in the Data Hub plugin.
Compute a simple aggregate over records and set a field on each record.
| Argument | Type | Required | Description |
|---|---|---|---|
op |
select | Yes | Operation |
Example:
{ op: 'aggregate', args: {
"op": "value"
} }
Count elements in an array or characters in a string.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
Example:
{ op: 'count', args: {
"source": "sourceField",
"target": "targetField"
} }
Remove duplicate values from an array field.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Target field path |
by |
string | No | Object key to use for uniqueness |
Example:
{ op: 'unique', args: {
"source": "sourceField"
} }
Flatten a nested array into a single-level array.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source path if not set |
depth |
number | No | How deep to flatten (default: 1) |
Example:
{ op: 'flatten', args: {
"source": "sourceField"
} }
Get the first element of an array.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source array path |
target |
string | Yes | Target field path |
Example:
{ op: 'first', args: {
"source": "sourceField",
"target": "targetField"
} }
Get the last element of an array.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source array path |
target |
string | Yes | Target field path |
Example:
{ op: 'last', args: {
"source": "sourceField",
"target": "targetField"
} }
Expand an array field into multiple records. Each array element becomes a separate record with optional parent field inheritance.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Path to the array to expand (e.g., |
mergeParent |
boolean | No | Include all parent fields in expanded records |
parentFields |
json | No | Map of target field names to source paths (e.g., { |
Example:
{ op: 'expand', args: {
"path": "sourceField"
} }
Join two datasets by matching key fields. Supports INNER, LEFT, RIGHT, and FULL OUTER join types.
| Argument | Type | Required | Description |
|---|---|---|---|
leftKey |
string | Yes | Field path in left (primary) records to join on |
rightKey |
string | Yes | Field path in right records to join on |
type |
select | Yes | Join type |
Example:
{ op: 'multiJoin', args: {
"leftKey": "value",
"rightKey": "value",
"type": "value"
} }
Resize images referenced in record fields (base64-encoded)
Example:
{ op: 'imageResize', args: {} }
Convert image format (JPEG, PNG, WebP, AVIF, GIF)
Example:
{ op: 'imageConvert', args: {} }
Generate PDF from HTML template with record data
Example:
{ op: 'pdfGenerate', args: {} }
Transform records via field mapping. Provide a JSON object of dst -> src dot-paths.
| Argument | Type | Required | Description |
|---|---|---|---|
mapping |
json | Yes | JSON object defining field mapping (target: source) |
passthrough |
boolean | No | If true, include fields not in mapping |
Example:
{ op: 'map', args: {
"mapping": "value"
} }
Set a static value at a specified path.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Dot notation path where to set the value |
value |
json | Yes | The value to set (any valid JSON) |
Example:
{ op: 'set', args: {
"path": "sourceField",
"value": "value"
} }
Remove a field at a specified path.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Dot notation path of the field to remove |
Example:
{ op: 'remove', args: {
"path": "sourceField"
} }
Rename a field from one path to another.
| Argument | Type | Required | Description |
|---|---|---|---|
from |
string | Yes | Source field path |
to |
string | Yes | Target field path |
Example:
{ op: 'rename', args: {
"from": "value",
"to": "value"
} }
Copy a field value to another path.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
Example:
{ op: 'copy', args: {
"source": "sourceField",
"target": "targetField"
} }
Render a string template and set it at target path.
| Argument | Type | Required | Description |
|---|---|---|---|
template |
string | Yes | Use ${path.to.field} to substitute values |
target |
string | Yes | Where to store the result |
missingAsEmpty |
boolean | No | Treat missing fields as empty strings |
Example:
{ op: 'template', args: {
"template": "value",
"target": "targetField"
} }
Generate a cryptographic hash (MD5, SHA1, SHA256, SHA512) of field value(s).
| Argument | Type | Required | Description |
|---|---|---|---|
source |
json | Yes | Single path string or array of paths to hash together |
target |
string | Yes | Path where the hash will be stored |
algorithm |
select | No | Default: sha256 |
encoding |
select | No | Default: hex |
Example:
{ op: 'hash', args: {
"source": "sourceField",
"target": "targetField"
} }
Generate a UUID for each record. Supports v4 (random) and v5 (namespace-based deterministic).
| Argument | Type | Required | Description |
|---|---|---|---|
target |
string | Yes | Path where the UUID will be stored |
version |
select | No | UUID version |
Example:
{ op: 'uuid', args: {
"target": "targetField"
} }
Mark records as invalid if required fields are missing.
| Argument | Type | Required | Description |
|---|---|---|---|
fields |
json | Yes | Required fields (JSON array) |
errorField |
string | No | Field to store validation errors |
Example:
{ op: 'validateRequired', args: {
"fields": "value"
} }
Validate field format using regex.
| Argument | Type | Required | Description |
|---|---|---|---|
field |
string | Yes | Field path |
pattern |
string | Yes | Regex pattern |
errorField |
string | No | Error output field |
errorMessage |
string | No | Error message |
Example:
{ op: 'validateFormat', args: {
"field": "value",
"pattern": "value"
} }
Format a date field to a string.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
format |
string | Yes | e.g. YYYY-MM-DD, DD/MM/YYYY HH:mm |
inputFormat |
string | No | If source is string, specify its format |
timezone |
string | No | e.g. UTC, Europe/London |
Example:
{ op: 'dateFormat', args: {
"source": "sourceField",
"target": "targetField",
"format": "value"
} }
Parse a string to a date.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
format |
string | Yes | Format of the source string |
timezone |
string | No | Timezone |
Example:
{ op: 'dateParse', args: {
"source": "sourceField",
"target": "targetField",
"format": "value"
} }
Add or subtract time from a date.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
amount |
number | Yes | Positive to add, negative to subtract |
unit |
select | Yes | Unit: seconds, minutes, hours, days, weeks, months, years |
Important: Unit strings must be plural:
"days","hours","minutes","seconds","weeks","months","years". Singular forms like"day"are not supported.
Example:
{ op: 'dateAdd', args: {
"source": "sourceField",
"target": "targetField",
"amount": 10,
"unit": "days"
} }
Calculate the difference between two dates in a specified unit.
| Argument | Type | Required | Description |
|---|---|---|---|
startDate |
string | Yes | Start date field path |
endDate |
string | Yes | End date field path |
target |
string | Yes | Target field path |
unit |
select | Yes | Result unit: seconds, minutes, hours, days, weeks, months, years |
Important: Unit strings must be plural:
"days","hours","minutes","seconds","weeks","months","years". Singular forms like"day"are not supported.
Example:
{ op: 'dateDiff', args: {
"startDate": "startDateField",
"endDate": "endDateField",
"target": "targetField",
"unit": "days"
} }
Set the current timestamp on a field. Useful for adding created/updated timestamps.
| Argument | Type | Required | Description |
|---|---|---|---|
target |
string | Yes | Target field path |
format |
select | No | Output format |
Example:
{ op: 'now', args: {
"target": "targetField"
} }
Lookup value from a map and set to target field.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
map |
json | Yes | Map (JSON object) |
target |
string | Yes | Target field path |
default |
string | No | Default value |
Example:
{ op: 'lookup', args: {
"source": "sourceField",
"map": "value",
"target": "targetField"
} }
Return the first non-null value from a list of field paths.
| Argument | Type | Required | Description |
|---|---|---|---|
paths |
json | Yes | Array of paths to check in order |
target |
string | Yes | Target field path |
default |
json | No | Value if all paths are null |
Example:
{ op: 'coalesce', args: {
"paths": "value",
"target": "targetField"
} }
Enrich or default fields on records.
| Argument | Type | Required | Description |
|---|---|---|---|
set |
json | No | JSON object of fields to set (dot paths allowed) |
defaults |
json | No | JSON object of fields to set if currently missing (dot paths allowed) |
Example:
{ op: 'enrich', args: {} }
Set a default value if field is null or undefined.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
value |
json | Yes | Default value (JSON) |
Example:
{ op: 'default', args: {
"path": "sourceField",
"value": "value"
} }
Enrich records by fetching data from external HTTP endpoints with caching, authentication, and error handling.
| Argument | Type | Required | Description |
|---|---|---|---|
url |
string | Yes | HTTP endpoint URL. Use for dynamic values. |
method |
select | No | HTTP Method |
target |
string | Yes | Field path to store the response data. |
responsePath |
string | No | JSON path to extract from response (optional). |
keyField |
string | No | Field to use as cache key. If not set, URL is used. |
default |
json | No | Value to use if lookup fails or returns 404. |
timeoutMs |
number | No | Request timeout in milliseconds. |
cacheTtlSec |
number | No | Cache time-to-live in seconds. Set to 0 to disable. |
headers |
json | No | Static HTTP headers as JSON object. |
bearerTokenSecretCode |
string | No | Secret code for Bearer token authentication. |
apiKeySecretCode |
string | No | Secret code for API key authentication. |
apiKeyHeader |
string | No | Header name for API key. |
basicAuthSecretCode |
string | No | Secret code for Basic auth (username:password). |
bodyField |
string | No | Field path for POST body (uses record value at this path). |
body |
json | No | Static POST body (JSON object). |
skipOn404 |
boolean | No | Skip record if endpoint returns 404. |
failOnError |
boolean | No | Fail pipeline if HTTP request fails. |
maxRetries |
number | No | Maximum retry attempts on transient errors. |
batchSize |
number | No | Process this many records in parallel (default: 50). |
rateLimitPerSecond |
number | No | Max requests per second per domain (default: 100). |
Example:
{ op: 'httpLookup', args: {
"url": "value",
"target": "targetField"
} }
Parse a JSON string field into an object.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source if not set |
Example:
{ op: 'parseJson', args: {
"source": "sourceField"
} }
Stringify an object field to a JSON string.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source if not set |
pretty |
boolean | No | Pretty print |
Example:
{ op: 'stringifyJson', args: {
"source": "sourceField"
} }
Pick specific fields from a record, discarding others.
| Argument | Type | Required | Description |
|---|---|---|---|
fields |
json | Yes | Array of field paths to keep |
Example:
{ op: 'pick', args: {
"fields": "value"
} }
Omit specific fields from a record.
| Argument | Type | Required | Description |
|---|---|---|---|
fields |
json | Yes | Array of field paths to remove |
Example:
{ op: 'omit', args: {
"fields": "value"
} }
Filter records by conditions. Action: keep or drop.
| Argument | Type | Required | Description |
|---|---|---|---|
conditions |
json | Yes | e.g. [{ field: |
Example:
{ op: 'when', args: {
"conditions": "value"
} }
Set a value based on a condition.
| Argument | Type | Required | Description |
|---|---|---|---|
condition |
json | Yes | e.g. { field: |
thenValue |
json | Yes | Then value (JSON) |
elseValue |
json | No | Else value (JSON) |
target |
string | Yes | Target field path |
Example:
{ op: 'ifThenElse', args: {
"condition": "value",
"thenValue": "value",
"target": "targetField"
} }
Set a value based on multiple conditions (like a switch statement).
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
cases |
json | Yes | Array of { value, result } objects |
default |
json | No | Default value (JSON) |
target |
string | Yes | Target field path |
Example:
{ op: 'switch', args: {
"source": "sourceField",
"cases": "value",
"target": "targetField"
} }
Filter out unchanged records using a stable hash stored in checkpoint. Keeps only changed/new based on idPath.
| Argument | Type | Required | Description |
|---|---|---|---|
idPath |
string | Yes | ID field path |
includePaths |
json | No | Subset of fields to hash; default is entire record |
excludePaths |
json | No | Fields to ignore when hashing |
Example:
{ op: 'deltaFilter', args: {
"idPath": "value"
} }
Perform math operations on numeric fields.
| Argument | Type | Required | Description |
|---|---|---|---|
operation |
select | Yes | Operation |
Example:
{ op: 'math', args: {
"operation": "value"
} }
Convert floats to minor units or re-map currency fields.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
decimals |
number | Yes | Decimals (e.g. 2) |
round |
select | No | Rounding |
Example:
{ op: 'currency', args: {
"source": "sourceField",
"target": "targetField",
"decimals": 10
} }
Convert units (e.g. g<->kg, cm<->m)
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
from |
select | Yes | From unit |
Example:
{ op: 'unit', args: {
"source": "sourceField",
"target": "targetField",
"from": "value"
} }
Convert a string field to a number.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Target field path |
default |
number | No | Value if conversion fails |
Example:
{ op: 'toNumber', args: {
"source": "sourceField"
} }
Convert a value to a string.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Target field path |
Example:
{ op: 'toString', args: {
"source": "sourceField"
} }
Parse a string to a number with locale-aware decimal/thousand separator handling.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source path |
locale |
string | No | e.g., |
default |
number | No | Value if parsing fails |
Example:
{ op: 'parseNumber', args: {
"source": "sourceField"
} }
Format a number as a localized string with optional currency or percent formatting.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
locale |
string | No | e.g., |
decimals |
number | No | Decimal places |
style |
select | No | Format style |
Example:
{ op: 'formatNumber', args: {
"source": "sourceField",
"target": "targetField"
} }
Convert a decimal amount to cents (minor currency units). Multiplies by 100 and rounds.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Field containing decimal amount (e.g., 19.99) |
target |
string | Yes | Field for cents amount (e.g., 1999) |
round |
select | No | Rounding |
Example:
{ op: 'toCents', args: {
"source": "sourceField",
"target": "targetField"
} }
Round a number to a specified number of decimal places.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source if not set |
decimals |
number | No | Default: 0 (round to integer) |
mode |
select | No | Rounding mode |
Example:
{ op: 'round', args: {
"source": "sourceField"
} }
Split a string field into an array by delimiter.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
delimiter |
string | Yes | Character(s) to split by |
trim |
boolean | No | Trim whitespace from each item |
Example:
{ op: 'split', args: {
"source": "sourceField",
"target": "targetField",
"delimiter": "value"
} }
Join an array field into a string.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
delimiter |
string | Yes | Character(s) to join with |
Example:
{ op: 'join', args: {
"source": "sourceField",
"target": "targetField",
"delimiter": "value"
} }
Trim whitespace from a string field.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
mode |
select | No | Mode |
Example:
{ op: 'trim', args: {
"path": "sourceField"
} }
Convert a string field to lowercase.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
Example:
{ op: 'lowercase', args: {
"path": "sourceField"
} }
Convert a string field to uppercase.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
Example:
{ op: 'uppercase', args: {
"path": "sourceField"
} }
Generate a URL-friendly slug from a string field.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
separator |
string | No | Default: hyphen (-) |
Example:
{ op: 'slugify', args: {
"source": "sourceField",
"target": "targetField"
} }
Concatenate multiple string fields into one.
| Argument | Type | Required | Description |
|---|---|---|---|
sources |
json | Yes | Array of field paths to concatenate |
target |
string | Yes | Target field path |
separator |
string | No | Optional separator between values |
Example:
{ op: 'concat', args: {
"sources": "value",
"target": "targetField"
} }
Replace text in a string field.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
search |
string | Yes | Search text |
replacement |
string | Yes | Replacement |
all |
boolean | No | Replace all occurrences |
Example:
{ op: 'replace', args: {
"path": "sourceField",
"search": "value",
"replacement": "value"
} }
Extract a value from a string field using a regular expression pattern with capture groups.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | Yes | Target field path |
pattern |
string | Yes | Regular expression pattern (without delimiters) |
group |
number | No | Group index to extract (0=full match, 1+=capture groups). Default: 1 |
flags |
string | No | e.g., |
Example:
{ op: 'extractRegex', args: {
"source": "sourceField",
"target": "targetField",
"pattern": "value"
} }
Replace values in a string field using a regular expression pattern.
| Argument | Type | Required | Description |
|---|---|---|---|
path |
string | Yes | Field path |
pattern |
string | Yes | Regular expression pattern (without delimiters) |
replacement |
string | Yes | Replacement string (use $1, $2 for capture groups) |
flags |
string | No | e.g., |
Example:
{ op: 'replaceRegex', args: {
"path": "sourceField",
"pattern": "value",
"replacement": "value"
} }
Remove HTML tags from a string field, preserving text content.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source if not set |
Example:
{ op: 'stripHtml', args: {
"source": "sourceField"
} }
Truncate a string to a maximum length, optionally adding a suffix.
| Argument | Type | Required | Description |
|---|---|---|---|
source |
string | Yes | Source field path |
target |
string | No | Defaults to source if not set |
length |
number | Yes | Maximum length |
suffix |
string | No | e.g., |
Example:
{ op: 'truncate', args: {
"source": "sourceField",
"length": 10
} }
Execute inline JavaScript code to transform records. Use for complex logic that cannot be expressed with standard operators.
| Argument | Type | Required | Description |
|---|---|---|---|
code |
code | Yes | JavaScript code to execute. In single-record mode: receives record, index, context. In batch mode: receives records, context. Must return the transformed result. |
batch |
boolean | No | If true, processes all records at once. If false (default), processes one record at a time. |
timeout |
number | No | Maximum execution time in milliseconds (default: 5000) |
failOnError |
boolean | No | If true, errors fail the entire step. If false, errors are logged and records skipped. |
context |
json | No | Optional JSON data passed to the script as context.data |
Example:
{ op: 'script', args: {
"code": "value"
} }
You can register custom operators using the SDK. See Custom Operators Guide for details.