vendure-data-hub-plugin

Operators Reference

Complete reference for all transform operators.

Table of Contents

Data Operators

Enrichment Operators

String Operators

Numeric Operators

Date Operators

Logic Operators

JSON Operators

Array Operators

Aggregation Operators

File Operators

Validation Operators

Script Operator

Data Operators

set

Set a static value at a path.

Arg Type Required Description
path string Yes Target field path
value any Yes Value to set (JSON) 
{ op: 'set', args: { path: 'enabled', value: true } }
{ op: 'set', args: { path: 'metadata.source', value: 'import' } }

copy

Copy a field value to another path.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path 
{ op: 'copy', args: { source: 'sku', target: 'externalId' } }

rename

Rename a field.

Arg Type Required Description
from string Yes Source field path
to string Yes Target field path 
{ op: 'rename', args: { from: 'product_name', to: 'name' } }

remove

Remove a field at path.

Arg Type Required Description
path string Yes Field path to remove 
{ op: 'remove', args: { path: 'temp_id' } }

map

Transform records via field mapping.

Arg Type Required Description
mapping json Yes JSON object defining field mapping (target: source)
passthrough boolean No If true, include fields not in mapping 
{ op: 'map', args: { mapping: { name: 'product_name', sku: 'product_sku', price: 'retail_price' } } }
{ op: 'map', args: { mapping: { name: 'title', price: 'cost' }, passthrough: true } }

template

Render a string template and set it at target path.

Arg Type Required Description
template string Yes Template with ${path} substitutions
target string Yes Target field path
missingAsEmpty boolean No Treat missing fields as empty string 
{ op: 'template', args: { template: 'https://store.com/${slug}', target: 'url' } }
{ op: 'template', args: { template: '${firstName} ${lastName}', target: 'fullName', missingAsEmpty: true } }

hash

Generate a deterministic cryptographic digest of field value(s).

Arg Type Required Description
source json Yes Single path string or array of paths to hash together
target string Yes Path where the hash will be stored
algorithm select No sha256, sha512 (default: sha256)
encoding select No hex or base64 (default: hex) 
{ op: 'hash', args: { source: 'data', target: 'checksum', algorithm: 'sha256' } }
{ op: 'hash', args: { source: ['sku', 'name', 'price'], target: 'contentHash', algorithm: 'sha256' } }
{ op: 'hash', args: { source: 'externalId', target: 'externalIdDigest', algorithm: 'sha512', encoding: 'base64' } }

This is not a password-hashing function: it has no salt or configurable work factor. Never use it to store passwords.

uuid

Generate a UUID.

Arg Type Required Description
target string Yes Target field path
version string No v4 (random) or v5 (namespace-based)
namespace string No For v5, a UUID or dns, url, oid, or x500; required with source
source string No Source field path used as the v5 name; required with namespace 
{ op: 'uuid', args: { target: 'id' } }
{ op: 'uuid', args: { target: 'productId', version: 'v4' } }
{ op: 'uuid', args: { target: 'stableId', version: 'v5', namespace: 'url', source: 'sku' } }

Enrichment Operators

lookup

Lookup value from a map and set to target field.

Arg Type Required Description
source string Yes Source field path
map object Yes Lookup map (JSON object)
target string Yes Target field path
default any No Default value if not found 
{ op: 'lookup', args: { source: 'status', map: { 'A': 'active', 'I': 'inactive' }, target: 'statusText', default: 'unknown' } }

enrich

Set or default fields on records.

Arg Type Required Description
set object No Fields to set (overwrites)
defaults object No Fields to set if currently missing 
{ op: 'enrich', args: { set: { source: 'import' }, defaults: { enabled: true } } }

coalesce

Return the first non-null value from paths.

Arg Type Required Description
paths array Yes Array of field paths to check
target string Yes Target field path
default any No Default value if all null 
{ op: 'coalesce', args: { paths: ['name', 'title', 'sku'], target: 'displayName', default: 'Unnamed' } }

default

Set a default value if field is null or undefined.

Arg Type Required Description
path string Yes Field path
value json Yes Default value 
{ op: 'default', args: { path: 'enabled', value: true } }
{ op: 'default', args: { path: 'stock', value: 0 } }

httpLookup

Enrich records by fetching data from external HTTP endpoints with caching, authentication, and error handling.

Arg Type Required Description
connectionCode connection No Saved HTTP connection; required for Secret-backed authentication and used to bind credentials and redirects to one origin
url string Yes HTTP endpoint URL. Use `` for dynamic values
method select No GET or POST (default: GET)
target string Yes Field path to store the response data
responsePath string No JSON path to extract from response
keyField string No Optional record value included in the opaque full-request cache identity
default json No Value to use if lookup fails or returns 404
timeoutMs number No Integer from 1 to 300000; default: 5000 ms
cacheTtlSec number No Integer from 0 to 604800 seconds; default: 300. Set to 0 to disable
headers json No Non-sensitive static string headers
bearerTokenSecretCode string No Secret code for Bearer token authentication
apiKeySecretCode string No Secret code for API key authentication
apiKeyHeader string No Header name for API key
basicAuthSecretCode string No Secret code for Basic auth (username:password)
bodyField string No Field path for POST body (uses record value at this path)
body json No Static POST body (JSON object)
skipOn404 boolean No Remove record from pipeline if endpoint returns 404 (see note below)
failOnError boolean No Report lookup failures as operator errors instead of writing default
maxRetries number No Integer from 0 to 10; default: 2 retries after the initial attempt
batchSize number No Parallel lookup concurrency from 1 to 500; default: 50
rateLimitPerSecond number No Integer from 1 to 10000 requests per second per domain; default: 100 
// Basic enrichment from external API
{ op: 'httpLookup', args: {
    url: 'https://api.example.com/products/',
    target: 'externalData',
    cacheTtlSec: 300,
} }

// With authentication and response extraction
{ op: 'httpLookup', args: {
    connectionCode: 'inventory-api',
    url: 'https://api.inventory.com/stock/',
    target: 'stockLevel',
    responsePath: 'data.available',
    bearerTokenSecretCode: 'inventory-api-key',
    default: 0,
} }

// POST request with body
{ op: 'httpLookup', args: {
    url: 'https://api.pricing.com/calculate',
    method: 'POST',
    bodyField: 'priceRequest',
    target: 'calculatedPrice',
    headers: { 'Content-Type': 'application/json' },
} }

The cache identity includes the execution namespace, resolved credential headers, URL, method, request body, response path, and optional keyField value; keyField does not replace the URL as the cache key. Circuit-breaker and process-local rate-limit state use a separate opaque key derived from the channel, endpoint, and resolved request headers. Failures or reservations from another channel or credential cannot open or throttle this lookup; lookups in the same channel using the same provider identity share the configured per-domain bucket. If those lookups declare different limits, the process uses the lower value until the idle bucket is discarded. Configure at most one of bearerTokenSecretCode, apiKeySecretCode, and basicAuthSecretCode. A Basic-auth secret must resolve to username:password. Missing or empty secrets fail before the request. Credential, cookie, signature, routing-control, and hop-by-hop headers are rejected from headers. Secret-backed authentication requires connectionCode referencing a saved HTTP-family connection with baseUrl. Public unauthenticated lookups may omit the connection. Cross-origin URLs and redirects fail before credentials can be sent.

skipOn404 behavior: When skipOn404: true and the enrichment HTTP lookup returns a 404, the record is removed from the pipeline (not passed through with original values). To preserve records when enrichment fails, use default to provide a fallback value instead.

String Operators

trim

Trim whitespace from a string field.

Arg Type Required Description
path string Yes Field path
mode string No both, start, or end 
{ op: 'trim', args: { path: 'name' } }
{ op: 'trim', args: { path: 'description', mode: 'both' } }

uppercase

Convert to uppercase.

Arg Type Required Description
path string Yes Field path 
{ op: 'uppercase', args: { path: 'sku' } }

lowercase

Convert to lowercase.

Arg Type Required Description
path string Yes Field path 
{ op: 'lowercase', args: { path: 'email' } }

slugify

Generate a URL-friendly slug.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
separator string No Separator character (default: -) 
{ op: 'slugify', args: { source: 'name', target: 'slug' } }
{ op: 'slugify', args: { source: 'title', target: 'urlPath', separator: '_' } }

split

Split string into array.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
delimiter string Yes Split delimiter
trim boolean No Trim whitespace from each item 
{ op: 'split', args: { source: 'tags', target: 'tagList', delimiter: ',' } }
{ op: 'split', args: { source: 'categories', target: 'categoryArray', delimiter: '|', trim: true } }

join

Join array into string.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
delimiter string Yes Join delimiter 
{ op: 'join', args: { source: 'path', target: 'breadcrumb', delimiter: ' > ' } }

concat

Concatenate multiple string fields.

Arg Type Required Description
sources array Yes Array of field paths
target string Yes Target field path
separator string No Separator between values 
{ op: 'concat', args: { sources: ['firstName', 'lastName'], target: 'fullName', separator: ' ' } }

replace

Replace text in a string field.

Arg Type Required Description
path string Yes Field path
search string Yes Text to find
replacement string Yes Replacement text
all boolean No Replace all occurrences 
{ op: 'replace', args: { path: 'description', search: '\n', replacement: '<br>', all: true } }

extractRegex

Extract text using regex capture groups.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
pattern string Yes Regex pattern with capture group (without delimiters)
group number No Capture group index (0=full match, 1+=capture groups). Default: 1
flags string No Regex flags (e.g., “i” for case-insensitive) 
{ op: 'extractRegex', args: { source: 'sku', pattern: '^([A-Z]+)-\\d+$', target: 'prefix' } }
{ op: 'extractRegex', args: { source: 'url', pattern: '/products/([^/]+)/', target: 'productSlug' } }
{ op: 'extractRegex', args: { source: 'text', pattern: '([a-z]+)', target: 'word', group: 1, flags: 'i' } }

replaceRegex

Replace text using regex pattern.

Arg Type Required Description
path string Yes Field path
pattern string Yes Regex pattern
replacement string Yes Replacement (supports $1, $2, etc.)
flags string No Regex flags (default: g) 
{ op: 'replaceRegex', args: { path: 'text', pattern: '\\s+', replacement: ' ' } }
{ op: 'replaceRegex', args: { path: 'html', pattern: '<[^>]+>', replacement: '' } }

stripHtml

Remove HTML tags from a string, preserving text content.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source) 
{ op: 'stripHtml', args: { source: 'description' } }
{ op: 'stripHtml', args: { source: 'htmlContent', target: 'plainText' } }

truncate

Truncate a string to a maximum length.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source)
length number Yes Maximum length
suffix string No Suffix when truncated (e.g., ...) 
{ op: 'truncate', args: { source: 'description', length: 100, suffix: '...' } }
{ op: 'truncate', args: { source: 'title', target: 'shortTitle', length: 50 } }

Numeric Operators

math

Perform math operations with flexible source/target fields.

Arg Type Required Description
operation string Yes add, subtract, multiply, divide, modulo, power, round, floor, ceil, abs
source string Yes Source field path
operand string/number No Value or field path (prefix with $ for field reference)
target string Yes Target field path
decimals number No Decimal places for result 
{ op: 'math', args: { operation: 'multiply', source: 'price', operand: '100', target: 'priceInCents' } }
{ op: 'math', args: { operation: 'divide', source: 'cents', operand: '100', target: 'dollars', decimals: 2 } }
{ op: 'math', args: { operation: 'round', source: 'value', target: 'rounded', decimals: 0 } }
{ op: 'math', args: { operation: 'abs', source: 'balance', target: 'absoluteBalance' } }

toNumber

Convert string to number.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source)
default number No Default value if conversion fails 
{ op: 'toNumber', args: { source: 'price' } }
{ op: 'toNumber', args: { source: 'quantity', target: 'qty', default: 0 } }

toString

Convert value to string.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source) 
{ op: 'toString', args: { source: 'id', target: 'idString' } }

currency

Convert floats to minor units.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
decimals number Yes Decimal places (e.g., 2 for cents)
round string No round, floor, or ceil 
{ op: 'currency', args: { source: 'price', target: 'priceInCents', decimals: 2, round: 'round' } }

unit

Convert units (g to kg, cm to m, etc).

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
from string Yes Source unit (g, kg, cm, m)
to string Yes Target unit 
{ op: 'unit', args: { source: 'weightGrams', target: 'weightKg', from: 'g', to: 'kg' } }
{ op: 'unit', args: { source: 'lengthCm', target: 'lengthM', from: 'cm', to: 'm' } }

parseNumber

Parse locale-formatted number strings.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source)
locale string No Locale for parsing (e.g., de-DE for 1.234,56)
default number No Value written when parsing fails; otherwise the operator writes null 
{ op: 'parseNumber', args: { source: 'priceEuro', target: 'price', locale: 'de-DE' } }
{ op: 'parseNumber', args: { source: 'amount', locale: 'fr-FR' } }

formatNumber

Format numbers with locale support.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
locale string No Output locale (e.g., en-US, de-DE); default: en-US
decimals number No Sets both minimum and maximum fraction digits; otherwise Intl style defaults apply
style select No decimal, currency, or percent; default: decimal
currency string No Currency code used with currency style; a missing code falls back to decimal formatting
useGrouping boolean No Use grouping separators; default: true 
{ op: 'formatNumber', args: { source: 'price', target: 'priceDisplay', style: 'currency', currency: 'USD' } }
{ op: 'formatNumber', args: { source: 'rate', target: 'ratePercent', style: 'percent', decimals: 1 } }
{ op: 'formatNumber', args: { source: 'amount', target: 'formatted', decimals: 2, useGrouping: true } }

toCents

Convert a decimal amount to cents (minor currency units).

Arg Type Required Description
source string Yes Source field path (decimal amount, e.g., 19.99)
target string Yes Target field path (cents, e.g., 1999)
round string No Rounding mode: round, floor, ceil 
{ op: 'toCents', args: { source: 'price', target: 'priceInCents' } }
{ op: 'toCents', args: { source: 'amount', target: 'centAmount', round: 'floor' } }

round

Round a number to a specified number of decimal places.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source)
decimals number No Decimal places (default: 0)
mode string No Rounding mode: round, floor, ceil 
{ op: 'round', args: { source: 'price', decimals: 2 } }
{ op: 'round', args: { source: 'weight', target: 'roundedWeight', decimals: 1, mode: 'ceil' } }

Date Operators

dateFormat

Format a date to a string.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
format string Yes Output format (e.g., YYYY-MM-DD)
inputFormat string No Input format if source is string

Custom formats use the exact UTC tokens YYYY, MM, DD, HH, mm, and ss and are limited to 128 characters. Separators must match and impossible calendar or clock values leave the target unchanged.

{ op: 'dateFormat', args: { source: 'createdAt', target: 'dateStr', format: 'YYYY-MM-DD' } }
{ op: 'dateFormat', args: { source: 'timestamp', target: 'formatted', format: 'DD/MM/YYYY HH:mm' } }

dateParse

Parse a string to a date.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
format string Yes Input format

The input format uses the same exact UTC token contract and 128-character limit. Invalid dates leave the target unchanged.

{ op: 'dateParse', args: { source: 'dateStr', target: 'date', format: 'MM/DD/YYYY' } }

dateAdd

Add or subtract time from a date.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path
amount number Yes Amount (negative to subtract)
unit string Yes seconds, minutes, hours, days, weeks, months, years

Important: Unit strings must be plural: "days", "hours", "minutes", "seconds", "weeks", "months", "years". Singular forms like "day" are not supported.

{ op: 'dateAdd', args: { source: 'orderDate', target: 'expiresAt', amount: 30, unit: 'days' } }
{ op: 'dateAdd', args: { source: 'createdAt', target: 'previousDay', amount: -1, unit: 'days' } }

dateDiff

Calculate the difference between two dates.

Arg Type Required Description
startDate string Yes Start date field path
endDate string Yes End date field path
target string Yes Target field path
unit select Yes seconds, minutes, hours, days, weeks, months, years
absolute boolean No Return absolute value (no negative numbers)

Important: Unit strings must be plural: "days", "hours", "minutes", "seconds", "weeks", "months", "years". Singular forms like "day" are not supported.

Months and years are elapsed-time approximations of 30.44 and 365.25 days, respectively; they are not calendar-month or anniversary calculations.

{ op: 'dateDiff', args: { startDate: 'createdAt', endDate: 'completedAt', target: 'durationDays', unit: 'days' } }
{ op: 'dateDiff', args: { startDate: 'orderDate', endDate: 'deliveryDate', target: 'deliveryHours', unit: 'hours', absolute: true } }

now

Set the current timestamp on a field.

Arg Type Required Description
target string Yes Target field path
format string No ISO, timestamp, date, datetime, or custom format 
{ op: 'now', args: { target: 'processedAt' } }
{ op: 'now', args: { target: 'dateOnly', format: 'date' } }
{ op: 'now', args: { target: 'formattedTime', format: 'YYYY-MM-DD HH:mm:ss' } }

Logic Operators

when

Filter records by conditions.

Arg Type Required Description
conditions array Yes Array of condition objects
action string Yes keep or drop

Condition operators: eq, ne, gt, gte, lt, lte, in, notIn, contains, notContains, startsWith, endsWith, regex, exists, notExists, isNull, isEmpty, isNotEmpty, matches (glob)

{ op: 'when', args: {
    conditions: [{ field: 'price', cmp: 'gt', value: 0 }],
    action: 'keep'
}}
{ op: 'when', args: {
    conditions: [{ field: 'status', cmp: 'in', value: ['active', 'pending'] }],
    action: 'keep'
}}

ifThenElse

Set a value based on a condition.

Arg Type Required Description
condition object Yes Condition object
thenValue any Yes Value if condition is true
elseValue any No Value if condition is false
target string Yes Target field path 
{ op: 'ifThenElse', args: {
    condition: { field: 'type', cmp: 'eq', value: 'digital' },
    thenValue: true,
    elseValue: false,
    target: 'isDigital',
}}

switch

Set a value based on multiple conditions.

Arg Type Required Description
source string Yes Source field to switch on
cases array Yes Array of { value, result } objects
default any No Default value
target string Yes Target field path 
{ op: 'switch', args: {
    source: 'status',
    cases: [
        { value: 'A', result: 'Active' },
        { value: 'I', result: 'Inactive' },
        { value: 'P', result: 'Pending' },
    ],
    default: 'Unknown',
    target: 'statusText',
}}

Comparison Operators

All 19 comparison operators used by when, ifThenElse, switch, and ROUTE step conditions:

Operator Description Example
eq Equal (strict ===) { field: 'status', cmp: 'eq', value: 'active' }
ne Not equal (strict !==) { field: 'status', cmp: 'ne', value: 'deleted' }
gt Greater than (numeric) { field: 'price', cmp: 'gt', value: 0 }
gte Greater than or equal (numeric) { field: 'stock', cmp: 'gte', value: 10 }
lt Less than (numeric) { field: 'price', cmp: 'lt', value: 1000 }
lte Less than or equal (numeric) { field: 'weight', cmp: 'lte', value: 50 }
in Value in array { field: 'status', cmp: 'in', value: ['active', 'pending'] }
notIn Value not in array { field: 'status', cmp: 'notIn', value: ['deleted', 'archived'] }
contains String contains substring { field: 'name', cmp: 'contains', value: 'widget' }
notContains String does not contain { field: 'name', cmp: 'notContains', value: 'test' }
startsWith String starts with { field: 'sku', cmp: 'startsWith', value: 'PRD-' }
endsWith String ends with { field: 'email', cmp: 'endsWith', value: '@example.com' }
regex Regular expression match { field: 'sku', cmp: 'regex', value: '^[A-Z]{3}-\\d+$' }
matches Glob pattern match { field: 'path', cmp: 'matches', value: '/products/**/*.json' }
exists Value is not null/undefined { field: 'email', cmp: 'exists', value: true }
notExists Value is null or undefined { field: 'deletedAt', cmp: 'notExists', value: true }
isNull Value is null or undefined { field: 'description', cmp: 'isNull', value: true }
isEmpty Empty string, null, undefined, or empty array { field: 'tags', cmp: 'isEmpty', value: true }
isNotEmpty Non-empty value (string, array, etc.) { field: 'name', cmp: 'isNotEmpty', value: true }

deltaFilter

Filter out unchanged records using stable hashing.

Arg Type Required Description
idPath string Yes Record identifier field
includePaths array No Fields to include in hash
excludePaths array No Fields to exclude from hash 
{ op: 'deltaFilter', args: { idPath: 'sku', includePaths: ['name', 'price', 'stock'] } }
{ op: 'deltaFilter', args: { idPath: 'id', excludePaths: ['updatedAt', 'createdAt'] } }

JSON Operators

pick

Keep only specified fields.

Arg Type Required Description
fields array Yes Array of field paths to keep 
{ op: 'pick', args: { fields: ['id', 'name', 'slug', 'sku', 'price'] } }

omit

Remove specified fields.

Arg Type Required Description
fields array Yes Array of field paths to remove 
{ op: 'omit', args: { fields: ['_internal', 'tempId', 'debug'] } }

parseJson

Parse a JSON string to an object.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source) 
{ op: 'parseJson', args: { source: 'metadataJson', target: 'metadata' } }

stringifyJson

Convert object to JSON string.

Arg Type Required Description
source string Yes Source field path
target string No Target field path (defaults to source)
pretty boolean No Pretty print output 
{ op: 'stringifyJson', args: { source: 'config', target: 'configJson', pretty: true } }

Array Operators

flatten

Flatten a nested array.

Arg Type Required Description
source string Yes Source array field path
target string No Target field path (defaults to source)
depth number No How deep to flatten (default: 1) 
{ op: 'flatten', args: { source: 'nestedCategories', target: 'categories' } }
{ op: 'flatten', args: { source: 'deepArray', depth: 2 } }

unique

Remove duplicate values from an array.

Arg Type Required Description
source string Yes Source array field path
target string No Target field path
by string No Object key for uniqueness (for arrays of objects) 
{ op: 'unique', args: { source: 'tags' } }
{ op: 'unique', args: { source: 'variants', target: 'uniqueVariants', by: 'sku' } }

first

Get the first element of an array.

Arg Type Required Description
source string Yes Source array path
target string Yes Target field path 
{ op: 'first', args: { source: 'images', target: 'featuredImage' } }

last

Get the last element of an array.

Arg Type Required Description
source string Yes Source array path
target string Yes Target field path 
{ op: 'last', args: { source: 'history', target: 'latestEntry' } }

count

Count array elements or characters in a string. Other value types produce 0.

Arg Type Required Description
source string Yes Source field path
target string Yes Target field path 
{ op: 'count', args: { source: 'variants', target: 'variantCount' } }

expand

Expand an array field into multiple records (one per element).

Arg Type Required Description
path string Yes Path to the array to expand (e.g., “variants” or “lines”)
mergeParent boolean No Include all parent fields in expanded records
parentFields json No Map of target field names to source paths (e.g., {"productId": "id", "productName": "name"}) 
{ op: 'expand', args: { path: 'variants' } }
{ op: 'expand', args: { path: 'lineItems', mergeParent: true } }
{ op: 'expand', args: { path: 'variants', parentFields: { productId: 'id', productName: 'name' } } }

This operator changes the record count. For example, 1 product with 3 variants becomes 3 records. Object elements become output records directly; primitive elements are written to _item. With the default mergeParent: false, a missing, non-array, or empty value emits no records. With mergeParent: true, that input emits the unchanged parent record. When expanding an object with mergeParent: true, its fields overwrite same-named parent fields.

Aggregation Operators

aggregate

Compute aggregates over records.

Arg Type Required Description
op select Yes count, sum, avg, min, max
source string No Source field path (required for sum/avg/min/max)
target string Yes Target field path 
{ op: 'aggregate', args: { op: 'count', target: 'totalRecords' } }
{ op: 'aggregate', args: { op: 'sum', source: 'quantity', target: 'totalQuantity' } }
{ op: 'aggregate', args: { op: 'avg', source: 'price', target: 'averagePrice' } }
{ op: 'aggregate', args: { op: 'min', source: 'price', target: 'lowestPrice' } }
{ op: 'aggregate', args: { op: 'max', source: 'price', target: 'highestPrice' } }

The aggregate is computed once across the current record set and the same result is written to target on every input record.

deduplicateRecords

Deduplicate the current record batch using a type-strict scalar key. The output position of each key is always its first-seen position, even when a later record wins.

Arg Type Required Description
key string Yes Scalar key field path
keep select No FIRST, LAST, LOWEST, or HIGHEST (default: FIRST)
priority string Conditional Finite numeric field path; required for LOWEST and HIGHEST 
{ op: 'deduplicateRecords', args: { key: 'sku' } }
{ op: 'deduplicateRecords', args: {
    key: 'sku',
    keep: 'LOWEST',
    priority: '_sourcePriority',
} }

String, boolean, and finite-number keys are distinct by both value and type. Missing, null, object, array, NaN, and infinite keys are not grouped and remain in the output. Ordered strategies fail if either competing record has a missing or non-finite priority.

multiJoin

Merge the current records with an inline right-side dataset by matching key fields. Supports one-to-many matches, INNER, LEFT, RIGHT, and FULL joins, optional field selection, and right-field prefixes.

Arg Type Required Description
leftKey string Yes Key field path in the current (left) dataset
rightKey string Yes Key field path in the right dataset
rightData array Yes Static array of right-side record objects
type select No Join type: INNER, LEFT, RIGHT, FULL (default: LEFT)
prefix string No Prefix for right fields; the operator adds an underscore
select array No Specific fields to include from the right dataset. If omitted or empty, all fields are included
maxOutputRecords number No Maximum emitted records; default 10000, maximum 100000 
// Inner join products with an inline price table
{ op: 'multiJoin', args: {
    leftKey: 'productId',
    rightKey: 'id',
    rightData: [
        { id: 'p1', price: 1999 },
        { id: 'p2', price: 2499 },
    ],
    type: 'INNER',
    maxOutputRecords: 10000,
} }

// Prefix "inv" produces fields such as inv_stock
{ op: 'multiJoin', args: {
    leftKey: 'sku',
    rightKey: 'sku',
    rightData: [
        { sku: 'SKU-1', stock: 12, warehouse: 'BER' },
    ],
    prefix: 'inv',
    select: ['stock'],
} }

rightData is inline configuration; the operator does not resolve another step’s output from a path. Model a dynamic multi-source join explicitly in the pipeline graph or provide a custom operator.

rightData accepts at most 10,000 objects. Join keys must resolve to finite numbers, strings, or booleans, and both the value and its type must match. Null, missing, array, object, NaN, and infinite key values never match; RIGHT and FULL joins preserve those right-side records as unmatched in their original order. Empty strings are valid string keys. Outer rows use the union of top-level fields found on each side and fill absent fields with null.

One-to-many matches can expand output quickly. The operator fails before it would emit more than maxOutputRecords; it never silently truncates. Use a right-field prefix whenever left and right records can share field names, because unprefixed right fields occupy the same output keys.

File Operators

imageResize

Resize base64-encoded images using sharp. Supports width, height, fit modes, output format, and quality settings.

Arg Type Required Description
sourceField string Yes Root record field containing the base64-encoded image
targetField string No Root record field for the result (defaults to source)
width number No Target width in pixels
height number No Target height in pixels
fit select No Resize fit mode: cover, contain, fill, inside, outside (default: cover)
format select No Output format: jpeg, png, webp, avif
quality number No Output quality (1-100). Default varies by format 
// Resize to specific dimensions
{ op: 'imageResize', args: { sourceField: 'photo', width: 800, height: 600, fit: 'cover' } }

// Resize and convert to WebP
{ op: 'imageResize', args: { sourceField: 'image', width: 400, format: 'webp', quality: 85 } }

// Resize to fit within bounds
{ op: 'imageResize', args: { sourceField: 'banner', targetField: 'thumbnail', width: 200, height: 200, fit: 'inside' } }

Requires the optional sharp package. If it is unavailable, the operator fails before processing records. Missing, empty, or invalid image values pass through unchanged.

imageConvert

Convert image format (JPEG, PNG, WebP, AVIF, GIF). Reads a base64-encoded image and re-encodes it in the target format.

Arg Type Required Description
sourceField string Yes Root record field containing the base64-encoded image
targetField string No Root record field for the result (defaults to source)
format select Yes Target format: jpeg, png, webp, avif, gif
quality number No Output quality (1-100). Default varies by format 
// Convert to WebP for smaller file sizes
{ op: 'imageConvert', args: { sourceField: 'image', format: 'webp', quality: 90 } }

// Convert to JPEG with quality setting
{ op: 'imageConvert', args: { sourceField: 'photo', targetField: 'jpegPhoto', format: 'jpeg', quality: 80 } }

// Convert to AVIF for modern browsers
{ op: 'imageConvert', args: { sourceField: 'image', format: 'avif', quality: 75 } }

Requires the optional sharp package. If it is unavailable, the operator fails before processing records. Missing, empty, or invalid image values pass through unchanged.

pdfGenerate

Generate a multi-page, plain-text PDF with nested-path `` placeholders using pdf-lib. Each record produces a base64-encoded PDF in the target field. Provide a static template or read one from templateField.

Arg Type Required Description
template string No Static text template with nested-path placeholders; used when templateField is absent or resolves to undefined
templateField string No Record field path containing the text template
targetField string Yes Field path for the generated base64-encoded PDF
pageSize select No Page size: A4, LETTER, A3 (default: A4)
orientation select No Page orientation: PORTRAIT, LANDSCAPE (default: PORTRAIT) 
// Generate a simple invoice PDF
{ op: 'pdfGenerate', args: {
    template: 'Invoice #\nCustomer: \nTotal: ',
    targetField: 'invoice_pdf',
} }

// Generate a product label in landscape
{ op: 'pdfGenerate', args: {
    template: '\nSKU: \n',
    targetField: 'label_pdf',
    pageSize: 'LETTER',
    orientation: 'LANDSCAPE',
} }

// Use a template stored in a record field
{ op: 'pdfGenerate', args: {
    templateField: 'htmlTemplate',
    targetField: 'document_pdf',
    pageSize: 'A3',
} }

This operator does not render HTML or CSS. Common block and line-break tags are converted to newlines, remaining HTML-like tags are stripped, and text is wrapped with standard Helvetica. Content continues on additional pages. pdf-lib is a direct runtime dependency. An individual record that cannot be rendered passes through without the target field and produces an operator error.

Validation Operators

validateRequired

Mark records as invalid if required fields are missing.

Arg Type Required Description
fields array Yes Array of required field paths
errorField string No Field to store validation errors 
{ op: 'validateRequired', args: { fields: ['sku', 'name', 'price'] } }

validateFormat

Validate field format using regex.

Arg Type Required Description
field string Yes Field path to validate
pattern string Yes Regex pattern
errorField string No Field to store error
errorMessage string No Custom error message 
{ op: 'validateFormat', args: { field: 'sku', pattern: '^[A-Z0-9-]+$', errorMessage: 'Invalid SKU format' } }
{ op: 'validateFormat', args: { field: 'email', pattern: '^[^@]+@[^@]+\\.[^@]+$' } }

Script Operator

Execute custom JavaScript for complex transformations that can’t be achieved with built-in operators.

script

Arg Type Required Description
code code Yes JavaScript code to execute. Receives record, index, context (single mode) or records, context (batch mode). Must return the transformed result.
batch boolean No If true, processes all records at once. If false (default), processes one record at a time.
timeout number No Integer from 1 to 300000 milliseconds (default: 5000)
failOnError boolean No If true, throw on an execution error. If false, log the error and preserve the original record(s)
context json No Optional JSON data passed to the script as context.data

Single Record Mode (default):

The code receives record, index, and context. Return the modified record (or null to exclude).

{ op: 'script', args: {
    code: `
        const margin = (record.price - record.cost) / record.price * 100;
        return { ...record, margin: Math.round(margin * 100) / 100 };
    `,
}}

// Filter out records by returning null
{ op: 'script', args: {
    code: `return record.stock > 0 ? record : null;`,
}}

// Use context data
{ op: 'script', args: {
    context: { taxRate: 0.2, currency: 'USD' },
    code: `
        const tax = record.price * context.data.taxRate;
        return { ...record, priceWithTax: record.price + tax, currency: context.data.currency };
    `,
}}

Batch Mode:

The code receives records array and context. Return the modified array.

{ op: 'script', args: {
    batch: true,
    code: `
        // Sort and rank all records
        const sorted = records.sort((a, b) => b.sales - a.sales);
        return sorted.map((r, i) => ({ ...r, rank: i + 1 }));
    `,
}}

// Calculate aggregate and add to each record
{ op: 'script', args: {
    batch: true,
    code: `
        const total = records.reduce((sum, r) => sum + r.amount, 0);
        return records.map(r => ({ ...r, percentOfTotal: r.amount / total * 100 }));
    `,
}}

Available in Script Context:

Array, Object, Math, JSON, and Date are limited safe wrappers, not unrestricted host constructors. console is not exposed.

When failOnError is false, a single-record failure preserves that original record and continues; a batch failure preserves the entire original batch.

Security: Scripts receive restricted globals and a VM timeout, but Node’s vm is not a security boundary for hostile code. Permit script-bearing pipelines only for trusted administrators, or disable script execution in the plugin security options. See SECURITY.md.

Quick Reference

Category Operators
Data (8) set, copy, rename, remove, map, template, hash, uuid
String (12) trim, uppercase, lowercase, slugify, split, join, concat, replace, extractRegex, replaceRegex, stripHtml, truncate
Numeric (9) math, toNumber, toString, currency, unit, parseNumber, formatNumber, toCents, round
Date (5) dateFormat, dateParse, dateAdd, dateDiff, now
Logic (4) when, ifThenElse, switch, deltaFilter
JSON (4) pick, omit, parseJson, stringifyJson
Enrichment (5) lookup, enrich, coalesce, default, httpLookup
Aggregation (9) aggregate, deduplicateRecords, multiJoin, flatten, count, unique, first, last, expand
File (3) imageResize, imageConvert, pdfGenerate
Validation (2) validateRequired, validateFormat
Advanced (1) script
This site is open source. Improve this page.