Common issues and solutions for Data Hub.
Symptoms: Run button does nothing or returns error
Possible causes:
RunDataHubPipeline permissionenabled statusSymptoms: Run completes but 0 records processed
Check:
dataPath points to the records array in the responseSymptoms: High error rate, records quarantined
Steps:
Symptoms: Records fail during transform step
Common causes:
Symptoms: Records reach load step but fail
Check:
Database connections:
API connections:
Symptoms: “Connection timeout” or “Request timeout”
Solutions:
Check the secret detail status and runtime source before changing the database row:
A stored ENV reference only proves that the variable name is syntactically valid. It does not prove the variable exists.
This means the configured key is missing, different from the encryption key, or the stored envelope is corrupt. Restore the correct key or replace the credential.
Runtime resolution rejects unencrypted database INLINE values. Configure a durable master key and enter a replacement value to create a new encrypted envelope. The UI never returns the old value.
A historical same-code database row can become the fallback after a code-first definition is removed. Before removal, inspect rows marked Code-first active and delete or deliberately migrate the inactive database row.
When configPath is set, missing, unreadable, unsupported, malformed, or non-object JSON/YAML is fatal by design. Fix the path relative to the process working directory, file permissions, extension, syntax, root object, duplicate codes, or invalid secret definition. The previous in-memory snapshot is never partially replaced.
Analyze:
Solutions:
Causes:
Solutions:
Symptoms: Jobs pile up, runs delayed
Solutions:
Adapter code doesn’t exist.
Solutions:
registerBuiltinAdapters is trueConnection code doesn’t exist.
Solutions:
Secret code doesn’t exist.
Solutions:
Pipeline JSON is malformed.
Solutions:
Missing permission.
Solutions:
DataHubPlugin.init({
debug: true,
})
Symptoms: A host-project Vendure migration fails or only part of its DDL is visible.
Solutions:
Run reviewed pending migrations with the current Vendure CLI:
npx vendure migrate -r
Revert only when the last migration’s down() method has been reviewed and
will not discard production data:
npx vendure migrate --revert
Do not enable synchronize or delete compiled migration files as a production
repair. See Database and Upgrade Migrations for generation,
backup, deployment, and rollback procedures.
Symptoms: “Too many connections” or “Connection pool timeout”
Solutions:
dbConnectionOptions: {
extra: {
max: 20, // Increase from default 10
}
}
Symptoms: “Deadlock detected” or “Lock wait timeout”
Solutions:
throughput: {
concurrency: 1, // Sequential processing
}
throughput: {
batchSize: 20, // Reduce lock contention
}
Symptoms: Pipeline doesn’t run when webhook called
Check:
POST https://your-domain.com/data-hub/webhook/your-path
query {
dataHubLogs(options: { take: 10 }) {
items {
id
level
message
stepKey
createdAt
}
totalItems
}
}
Symptoms: “Invalid signature” or “Unauthorized”
Solutions:
const crypto = require('crypto');
const secret = 'your-secret';
const payload = JSON.stringify(requestBody);
const signature = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
Symptoms: Same webhook processed multiple times
Solutions:
trigger: {
type: 'WEBHOOK',
authentication: 'HMAC',
secretCode: 'webhook-secret',
requireIdempotencyKey: true,
idempotencyKeyHeader: 'X-Request-ID',
}
Symptoms: Pipeline doesn’t execute at scheduled time
Check:
# Test cron expression
# Use online cron validator
0 2 * * * # Valid: 2 AM daily
trigger: {
type: 'SCHEDULE',
cron: '0 2 * * *',
timezone: 'America/New_York', // Explicit timezone
}
# Check logs for scheduler
pm2 logs vendure | grep "SchedulerService"
Symptoms: Pipeline runs at unexpected times
Solutions:
date
timedatectl # Linux
timezone: 'UTC' // Always use explicit timezone
Symptoms: Vendure event occurs but pipeline doesn’t run
Check:
event: 'ProductEvent' // Must match Vendure event class
The value must be one of the event class names offered by the Dashboard.
Wildcards, action suffixes such as .updated, and trigger-level filter
fields are rejected. Filter seeded records in a downstream pipeline step.
data_hub_event_trigger_outboxattempts, and lastError for enqueue failuresdata-hub.event-trigger-outbox and data-hub.runquery {
dataHubLogs(options: { take: 10 }) {
items {
id
level
message
stepKey
createdAt
}
totalItems
}
}
Symptoms: The extractor logs that an uploaded file is missing or empty and returns no records.
Solutions:
Confirm the step uses the format-specific adapter and a Data Hub file ID:
.extract('parse-csv', {
adapterCode: 'csv',
fileId: 'uploaded-file-id',
hasHeader: true,
})
POST /data-hub/upload. Copy file.id from the response; a filename or server path is not a valid fileId.GET /data-hub/files to verify that the ID still exists. Uploaded files can expire according to the configured retention policy.ManageDataHubFiles to upload and ReadDataHubFiles to list or inspect files.Symptoms: The extractor returns no records or logs a CSV, JSON, XML, or spreadsheet parse error.
Checks:
csv, json, xml, or xlsx.delimiter and hasHeader. TSV has no separate adapter; set delimiter: '\t' on the CSV adapter.itemsPath points to the array of records.recordPath points to the repeated record elements.sheetName and whether the sheet contains a header row.Symptoms: Out-of-memory errors with large uploads.
Uploaded files are currently parsed into memory before downstream batches execute. Reducing throughput.batchSize can reduce downstream processing pressure but does not make parsing streaming.
resetCheckpoint: true on the extractor to start from the beginning.Symptoms: 429 Too Many Requests errors
Solutions:
throughput: {
rateLimitRps: 5, // 5 requests per second
}
throughput: {
concurrency: 1, // Sequential requests
}
errorHandling: {
maxRetries: 5,
retryDelayMs: 2000,
backoffMultiplier: 2, // Exponential backoff
}
Symptoms: “Cannot read property of undefined”
Solutions:
dataPath: 'data.items' // Must match response structure
curl -X GET https://api.example.com/products \
-H "Authorization: Bearer token"
operators: [
{ op: 'default', args: { path: 'items', value: [] } }
]
Symptoms: “UNABLE_TO_VERIFY_LEAF_SIGNATURE” or “CERT_HAS_EXPIRED”
Solutions:
# Ubuntu/Debian
sudo apt-get update ca-certificates
# macOS
brew upgrade openssl
ssl: {
enabled: true,
rejectUnauthorized: true,
caSecretCode: 'database-ca',
}
Store the PEM CA certificate under the referenced Secret Code. For mutual
TLS, configure both ssl.certSecretCode and ssl.keySecretCode. For generic
Node.js HTTPS clients that use the system trust store, provide a scoped
process trust bundle through NODE_EXTRA_CA_CERTS at process startup.
Never set NODE_TLS_REJECT_UNAUTHORIZED=0. It disables certificate verification
for every TLS client in the process, including unrelated integrations.
Symptoms: Search results don’t match database
Solutions:
Data Hub does not expose a rebuildDataHubSearchIndex mutation. Do not clear
an external index unless the replacement pipeline and rollback plan have been
tested.
Symptoms: “Index error” or documents not appearing
Check:
batchSize: 500 // Reduce if failing
Symptoms: Pipeline runs through gate without pausing
Check:
approvalType: 'MANUAL' // Requires manual approval
query {
dataHubLogs(options: { take: 10 }) {
items {
id
level
message
stepKey
createdAt
}
totalItems
}
}
Symptoms: Gate doesn’t auto-approve after timeout
Solutions:
timeoutSeconds: 3600 // 1 hour
timeoutSeconds must be an integer between 1 and 31,536,000 and is required
when approvalType is TIMEOUT.
gateStepKey, gateTimeoutAt,
gateTimeoutLeaseToken, and gateTimeoutLeaseExpiresAt plus both status
indexes to data_hub_pipeline_rundataHubPipelineRunSymptoms: “Adapter not found” for custom adapter
Check:
DataHubPlugin.init({
adapters: [myCustomAdapter],
})
code: 'my-custom-adapter' // Exact match required
npm run build
import { myAdapter } from './adapters/my-adapter';
Symptoms: Transform step fails with custom operator
Debug:
applyOne(record, config, helpers) {
console.log('Input:', record);
// ... operator logic ...
console.log('Output:', result);
return result;
}
const result = myOperator.applyOne(
{ test: 'data' },
{ /* config */ },
helpers
);
expect(result).toEqual({ /* expected */ });
Symptoms: Memory usage grows over time
Tools:
node --inspect server.js
# Open chrome://inspect
# Take heap snapshots
setInterval(() => {
const used = process.memoryUsage();
console.log('Memory:', Math.round(used.heapUsed / 1024 / 1024), 'MB');
}, 60000);
npm install -g clinic
clinic doctor -- node server.js
// Bad
eventEmitter.on('event', handler);
// Good
const handler = () => { /* ... */ };
eventEmitter.once('event', handler);
// Or: eventEmitter.removeListener('event', handler);
// Bad
const cache = new Map(); // Never cleared
// Good
const cache = new LRU({ max: 1000 }); // Bounded
// Bad
setInterval(fn, 1000);
// Good
const timer = setInterval(fn, 1000);
// Later: clearInterval(timer);
DataHubPlugin.init({
logging: {
level: 'DEBUG', // DEBUG, INFO, WARN, ERROR
logQueries: true,
logSteps: true,
},
})
.hooks({
AFTER_EXTRACT: [{
type: 'INTERCEPTOR',
name: 'Debug log',
code: `
console.log('Extracted records:', records.length);
console.log('Sample:', records[0]);
return records;
`,
}],
})
mutation {
startDataHubPipelineRun(pipelineId: "pipeline-id") {
id
status
}
}
-- Check recent runs
SELECT id, status, started_at, records_processed
FROM data_hub_pipeline_run
ORDER BY started_at DESC
LIMIT 10;
-- Check errors
SELECT * FROM data_hub_record_error
WHERE run_id = 'run-id'
LIMIT 10;
-- Check checkpoints
SELECT * FROM data_hub_checkpoint
WHERE pipeline_id = 'pipeline-id';
Hook contexts are recreated at every stage, so values written to context in a
before hook are not available to its matching after hook. Use persisted log
timestamps and run analytics for duration measurements; hooks can add boundary
markers:
.hooks({
BEFORE_TRANSFORM: [{
type: 'LOG',
level: 'INFO',
message: 'Transform step started',
}],
AFTER_TRANSFORM: [{
type: 'LOG',
level: 'INFO',
message: 'Transform step completed',
}],
})
Cancel via GraphQL:
mutation {
cancelDataHubPipelineRun(id: "run-id") {
id
status
}
}
A forced process termination can leave a run marked RUNNING; it is not a
substitute for cancellation and recovery.
DATAHUB_MASTER_KEY as one release unit.Reinstall exactly from the restored lockfile when required:
npm ci
-- View queue
SELECT * FROM job_queue
WHERE queue_name = 'data-hub.run'
AND state = 'PENDING';
-- Clear stuck jobs (use with caution)
DELETE FROM job_queue
WHERE queue_name = 'data-hub.run'
AND state = 'PENDING'
AND created_at < NOW() - INTERVAL '1 hour';
If you can’t resolve an issue:
When reporting issues, include:
# System info
node --version
npm --version
npx vendure version
# Plugin version
npm list @oronts/vendure-data-hub-plugin
# Database version
psql --version
# Recent logs
pm2 logs vendure --lines 100
# Pipeline configuration (sanitized)
# Export pipeline as JSON, remove sensitive data