Job Lifecycle¶

Every MultiFlexi job follows a well-defined lifecycle ensuring reliable execution, comprehensive logging, and automatic artifact preservation.

Target Audience: Users, Developers Difficulty: Intermediate Prerequisites: Understanding of Data Model

Lifecycle Overview ¶

A job progresses through distinct phases from creation to completion:

┌──────────────┐
│  Created     │  Scheduler creates job record
└──────┬───────┘
       │
┌──────▼───────┐
│  Scheduled   │  Entry added to schedule table
└──────┬───────┘
       │
┌──────▼───────┐
│   Queued     │  Executor detects ready job
└──────┬───────┘
       │
┌──────▼───────┐
│   Running    │  Application executes
└──────┬───────┘
       │
┌──────▼───────┐
│  Completed   │  Results recorded
└──────┬───────┘
       │
┌──────▼───────┐
│  Artifacts   │  Outputs preserved
│  Preserved   │
└──────────────┘

Phase 1: Job Creation ¶

Trigger Points:

Scheduler daemon determines RunTemplate is due
User clicks “Execute Now” in web interface
API call: POST /api/.../job/
CLI command: multiflexi-cli job:create --template <id>

What Happens:

New job record inserted with:
- runtemplate_id: Parent template reference
- company_id, app_id: Company and application
- executor: Execution environment (Native, Docker, etc.)
- env: Serialized environment variables
- launched_by: User ID who triggered execution
Environment assembly from multiple sources:
- Application default configuration
- Company credential values
- RunTemplate custom overrides
Job status set to “pending”

Database State:

INSERT INTO job (runtemplate_id, company_id, app_id, executor, env, launched_by)
VALUES (42, 15, 8, 'Native', '{"API_KEY":"..."}', 1);

Phase 2: Job Scheduling ¶

Triggered By: Job creation

What Happens:

Entry added to schedule table:
- job_id: Reference to job
- after: Execution timestamp (NOW for adhoc, calculated for scheduled)
RunTemplate updated:
- next_schedule set to NULL (prevents duplicate scheduling)
- last_schedule updated to current timestamp

Database State:

INSERT INTO schedule (job_id, after)
VALUES (12345, '2026-01-21 18:00:00');

UPDATE runtemplate
SET next_schedule = NULL, last_schedule = NOW()
WHERE id = 42;

Monitoring:

Zabbix receives “prepared” phase event (if enabled)
OpenTelemetry span created for job lifecycle (if enabled)

Phase 3: Job Queuing ¶

Triggered By: Executor daemon polling cycle

Executor Daemon Behavior:

Polls schedule table every N seconds (configurable via MULTIFLEXI_CYCLE_PAUSE)
Query: SELECT * FROM schedule WHERE after < NOW() ORDER BY after ASC
Respects parallel execution limit (MULTIFLEXI_MAX_PARALLEL)

What Happens:

Executor loads job record from database
Deserializes environment variables
Determines execution method based on executor field
Job state changes to “queued” (internal state)

Executor Types:

Native: Symfony Process component on host
Podman: Container execution with podman run
Docker: Container execution with docker run
Kubernetes: Pod deployment
Azure: Cloud function invocation

Phase 4: Job Execution ¶

What Happens:

Pre-execution (runBegin):
- Job begin timestamp recorded
- Process ID (PID) captured
- Status set to “running”
- Zabbix receives “running” phase event
Execution (launchJob):
- Command constructed: executable + cmdparams
- Environment variables injected
- Process launched
- Output streams captured in real-time:
  - stdout → buffered
  - stderr → buffered
- Progress monitoring (for long-running jobs)
Process Monitoring:
- Exit code tracked
- Execution duration calculated
- Resource usage monitored (memory, CPU)

Example Native Execution:

$process = new Process(
    ['/usr/bin/bank-importer', '--format=json'],
    null,  // working directory
    $environmentVars  // from job.env
);
$process->start();
$exitCode = $process->wait();
$stdout = $process->getOutput();
$stderr = $process->getErrorOutput();

Example Docker Execution:

docker run --rm \
  -e API_KEY=xxx \
  -e DATABASE_URL=yyy \
  vitexsoftware/bank-importer:latest \
  --format=json

Phase 5: Job Completion ¶

Triggered By: Process termination

What Happens (runEnd):

Result Capture:
- exitcode: Process exit status (0 = success, non-zero = failure)
- stdout: Complete standard output
- stderr: Complete standard error
- end: Completion timestamp
- pid: Process identifier
Database Updates:

UPDATE job
SET exitcode = 0,
    stdout = 'Bank import completed: 42 transactions',
    stderr = '',
    begin = '2026-01-21 18:00:05',
    end = '2026-01-21 18:00:23',
    pid = 12345
WHERE id = 12345;

RunTemplate Statistics:

UPDATE runtemplate
SET successfull_jobs_count = successfull_jobs_count + 1
WHERE id = 42;

Schedule Cleanup:

DELETE FROM schedule WHERE job_id = 12345;

Next Schedule Calculation:

UPDATE runtemplate
SET next_schedule = '2026-01-22 18:00:00'  -- Next day for daily jobs
WHERE id = 42;

Monitoring:

Zabbix receives “completed” phase with exitcode
OpenTelemetry span closed with final status
SQL audit log entry created

Phase 6: Artifact Preservation ¶

Automatically Executed: After job completion

MultiFlexi preserves all output from a completed job — both standard streams and any output files the application produces.

What Gets Preserved:

Standard Output (stdout.txt):
- Content-Type: text/plain
- Contains complete stdout stream
- Useful for debugging and result verification
Standard Error (stderr.txt):
- Content-Type: text/plain
- Contains complete stderr stream
- Essential for troubleshooting failures
Application Result Files (multi-file):

MultiFlexi collects output files from two sources, in order:
1. Legacy RESULT_FILE variable — If the application defines a RESULT_FILE environment variable, the file at that path is collected (backward-compatible with older applications).
2. Artifact definitions from the app manifest — Each entry in the artifacts array of the application JSON contains a path field that is a regex pattern. After the job finishes, MultiFlexi scans the temp directory and stores every file whose name matches at least one of these patterns.
For every matched file:
- MIME type is auto-detected
- Original filename is preserved
- Content is stored in the artifacts table
- Duplicates are skipped (a file already collected via RESULT_FILE will not be stored again)

Artifact Collection Flow:

Job finishes
    │
    ├── stdout / stderr → stored as artifacts
    │
    ├── RESULT_FILE env var set?
    │       └── Yes → resolve path → store file (if it exists)
    │
    └── app_artifacts definitions exist?
            └── Yes → for each definition:
                    scan temp dir with path regex
                        └── store every matching file

Example Artifact Definition (in application JSON):

"artifacts": [
  {
    "name": {"en": "JSON Report", "cs": "JSON zpráva"},
    "type": "file",
    "path": ".*\\.json$",
    "description": {"en": "Output report in JSON format"}
  },
  {
    "name": {"en": "CSV Export", "cs": "CSV export"},
    "type": "file",
    "path": ".*\\.csv$",
    "description": {"en": "Exported data in CSV format"}
  }
]

With this definition, MultiFlexi will automatically collect all .json and .csv files from the temp directory after every job execution.

Database Storage:

-- Artifact definitions (imported from app JSON, stored per-application)
INSERT INTO app_artifacts (app_id, path, type)
VALUES (8, '.*\.json$', 'file'),
       (8, '.*\.csv$',  'file');

INSERT INTO app_artifact_translations (app_artifact_id, lang, name, description)
VALUES (1, 'en', 'JSON Report', 'Output report in JSON format'),
       (1, 'cs', 'JSON zpráva', NULL),
       (2, 'en', 'CSV Export',  'Exported data in CSV format');

-- Collected artifacts (stored per-job after execution)
INSERT INTO artifacts (job_id, filename, content_type, artifact, note, created_at)
VALUES
  (12345, 'stdout.txt', 'text/plain', 'Bank import completed...', 'Standard output', NOW()),
  (12345, 'stderr.txt', 'text/plain', '', 'Standard error', NOW()),
  (12345, 'import-results.json', 'application/json', '{...}', 'Result file', NOW()),
  (12345, 'transactions.csv', 'text/csv', 'id,amount,...', 'Result file', NOW());

Access Methods:

Web UI: Job detail page → Artifacts section → Download button
API: GET /api/.../artifact/<id>
CLI: multiflexi-cli artifact:get <id> --output file.txt

Job Status Indicators ¶

Exit Code Interpretation ¶

Exit Code	Status	Meaning
0	✅ Success	Job completed successfully
1	⚠️ Warning	Minor issues, check logs
127	🔵 Not Found	Command/binary not found
255	❌ Failure	Critical error, job failed
-1	🔄 Retry	Transient error, retry recommended

Visual Indicators ¶

Web interface displays color-coded status:

Green (bg-success): Exit code 0
Yellow (bg-warning): Exit code 1
Red (bg-danger): Exit code 255
Blue (bg-primary): Exit code 127
Cyan (bg-info): Exit code -1

Job Persistence ¶

Retention Policy:

Jobs are never automatically deleted
Full execution history retained indefinitely (by default)
Supports compliance and debugging requirements

Manual Cleanup:

# Delete jobs older than 90 days
multiflexi-cli job:cleanup --days=90

GDPR Compliance:

Data retention policies can be configured per application or company. See GDPR Compliance for details.

Error Handling ¶

Common Failure Scenarios ¶

Application Binary Not Found (Exit 127):

Cause: Executable not installed or incorrect path
Solution: Verify application package installed: apt list --installed | grep multiflexi-<app>

Permission Denied (Exit 1 or 126):

Cause: Insufficient file/directory permissions
Solution: Check executor daemon runs as correct user (multiflexi)

Timeout:

Cause: Application exceeds configured timeout
Solution: Increase timeout in application configuration

Database Connection Failed:

Cause: Credential invalid or database unreachable
Solution: Verify credential values, test database connectivity

Retry Strategies ¶

Manual Retry:

multiflexi-cli job:retry <job-id>

Automatic Retry:

Configure in RunTemplate:

max_retries: Number of retry attempts
retry_delay: Seconds between retries
retry_on_exit_codes: Only retry for specific exit codes

Monitoring and Observability ¶

Real-Time Monitoring ¶

Track job execution in real-time:

Web UI Dashboard: Shows currently running jobs
Job Detail Page: Real-time status updates (requires page refresh)
System Logs: journalctl -u multiflexi-executor -f

Performance Metrics ¶

Key metrics to monitor:

Execution Duration: Time from begin to end
Queue Wait Time: Time from scheduled to begin
Success Rate: Percentage of jobs with exitcode 0
Failure Patterns: Common exit codes for failures

Zabbix Integration ¶

Metrics sent to Zabbix:

multiflexi.job.phase[<job_id>]: Current phase (prepared, running, completed)
multiflexi.job.exitcode[<job_id>]: Final exit code
multiflexi.job.duration[<job_id>]: Execution time in seconds

See Zabbix Integration for configuration.

OpenTelemetry Tracing ¶

Distributed tracing spans:

Span Name: job.execute
Attributes: job_id, runtemplate_id, company_id, app_id, executor
Events: phase transitions (prepared → running → completed)

See OpenTelemetry Integration for configuration.

Best Practices ¶

Application Design ¶

Idempotent Operations: Jobs should be safely re-runnable
Clear Exit Codes: Use 0 for success, non-zero for specific failures
Structured Output: Emit JSON/XML for easier parsing
Progress Logging: Log milestones to stdout for monitoring

Resource Management ¶

Memory Limits: Configure MULTIFLEXI_MEMORY_LIMIT_MB for executor
Parallel Execution: Set MULTIFLEXI_MAX_PARALLEL to prevent overload
Cleanup: Applications should clean up temporary files

Artifact Management ¶

File Organization: Use descriptive filenames
Size Awareness: Monitor artifact storage growth
Retention: Implement cleanup for old artifacts if needed

Troubleshooting ¶

Review Artifacts: Always check stdout.txt and stderr.txt first
Check Credentials: Verify credential assignment and values
Test Locally: Run application manually with same environment
Enable Debug Logging: Set application debug flags in RunTemplate config