MultiFlexi Architecture

The MultiFlexi ecosystem is composed of several interrelated projects, each fulfilling a specific role in the overall architecture. Below is an overview of the main member projects and their relationships:

Core Components

Project Relationships

  • The core library is a dependency for the CLI, server, and executor.

  • The database project provides schema and migrations for all components that require persistent storage.

  • The CLI and web both interact with the server (via REST API), database, and core library, but serve different user interfaces.

  • The executor is managed by the server and is responsible for running jobs in a secure and isolated manner.

  • The ansible-collection is used to automate deployment and configuration of all components.

  • The all meta-repo is used for orchestration, CI/CD, and as a reference for the complete MultiFlexi stack.

This modular architecture allows for flexible deployment, scaling, and maintenance of the MultiFlexi platform.

MultiFlexi Components Relationship Diagram

MultiFlexi project relationships schema

Scheduling and Execution Process

MultiFlexi uses a sophisticated scheduling system that handles job creation, queuing, and execution across multiple environments. Starting with version 2.x, the system uses dedicated systemd services instead of traditional cron jobs.

System Services

MultiFlexi 2.x+ operates with two dedicated systemd services:

  • multiflexi-scheduler.service: Continuously runs the scheduling daemon that creates job records

  • multiflexi-executor.service: Continuously polls for scheduled jobs and executes them

Both services run as daemon processes under the multiflexi user and are configured via /etc/multiflexi/multiflexi.env.

Phase 1: Scheduling (multiflexi-scheduler daemon)

The scheduling daemon (/usr/lib/multiflexi-scheduler/daemon.php) runs continuously in a loop and:

  • Loads all active companies from the database

  • Retrieves RunTemplates that are ready to be scheduled (active=true, next_schedule=null, interv != 'n')

  • Determines the scheduling interval:

    • Custom cron expressions (interv='c'): Uses the cron expression directly from the cron field

    • Standard intervals (i/h/d/w/m/y): Converts shorthand notation to full cron expressions using the $intervCron mapping:

      • i (minutely) → * * * * *

      • h (hourly) → 0 * * * *

      • d (daily) → 0 0 * * *

      • w (weekly) → 0 0 * * 1 (Monday)

      • m (monthly) → 0 0 1 * *

      • y (yearly) → 0 0 1 1 *

  • Parses the cron expression using the dragonmantank/cron-expression library

  • Calculates the next run time with getNextRunDate()

  • Applies optional startup delay if configured

  • Creates a new Job record via prepareJob()

Phase 2: Job Creation

During job creation:

  • A new record is inserted into the job table with:

    • runtemplate_id: Reference to the RunTemplate

    • company_id and app_id: Company and application identifiers

    • schedule: Calculated execution timestamp

    • executor: Selected executor type (Native, Podman, Docker, etc.)

    • env: Serialized environment variables from merged configurations

  • Environment variables are assembled from multiple sources:

    • Application-level configuration

    • Company-specific settings

    • Custom overrides from the RunTemplate

  • The job is scheduled by inserting a record into the schedule table with the after timestamp

  • The RunTemplate’s next_schedule field is updated

  • Optional monitoring metrics are sent to Zabbix (phase: ‘prepared’)

  • Optional OpenTelemetry trace spans are created for job lifecycle tracking

Phase 3: Job Execution (multiflexi-executor daemon)

The execution daemon (/usr/lib/multiflexi-executor/daemon.php) runs continuously and:

  • Periodically polls the schedule table for jobs where after < NOW() (default: every 10 seconds, configurable via MULTIFLEXI_CYCLE_PAUSE)

  • Loads the associated Job record and deserializes the environment

  • Selects the appropriate executor based on the job configuration:

    • Native Executor: Uses Symfony Process to run commands directly on the host machine with full environment variable support

    • Podman Executor: Runs the job in an isolated container with podman run

    • Docker Executor: Similar to Podman but using Docker containers

    • Azure/Kubernetes: Cloud-based execution in managed environments

  • The executor:

    • Constructs the command line from executable + cmdparams

    • Launches the process with the prepared environment

    • Captures real-time stdout and stderr output

    • Tracks the process ID (PID)

    • Monitors execution progress

  • After execution completes:

    • Updates the job table with results:

      • exitcode: Process exit status

      • stdout and stderr: Captured output

      • begin and end: Execution timestamps

      • pid: Process identifier

    • Updates RunTemplate statistics (successfull_jobs_count or failed_jobs_count)

    • Deletes the entry from the schedule table

    • Sets next_schedule=NULL and updates last_schedule

    • Sends final metrics to Zabbix and OpenTelemetry

    • Logs the execution to the SQL logging system

MultiFlexi Scheduling and Execution Process

Complete flow of job scheduling, creation, and execution in MultiFlexi

Daemon Configuration

The daemon services can be controlled via environment variables in /etc/multiflexi/multiflexi.env:

Scheduler Daemon:

  • MULTIFLEXI_DAEMONIZE=true: Run continuously (default) or exit after one cycle

Executor Daemon:

  • MULTIFLEXI_DAEMONIZE=true: Run continuously (default) or exit after one cycle

  • MULTIFLEXI_CYCLE_PAUSE=10: Seconds between polling cycles (default: 10)

  • MULTIFLEXI_MEMORY_LIMIT_MB=1800: Soft memory limit in MB for graceful shutdown before OOM

  • MULTIFLEXI_MAX_PARALLEL=4: Maximum concurrent jobs (requires pcntl extension; 0 for unlimited)

The executor daemon includes:

  • Memory monitoring: Proactively exits when approaching the memory limit to prevent OOM kills

  • Database resilience: Handles transient database failures with retries; exits on permanent errors (auth, missing database)

  • Graceful cleanup: Ensures proper shutdown with logging on exit

Key Design Principles

  1. Continuous Operation: Both scheduler and executor run as systemd services, eliminating dependency on system cron

  2. Separation of Concerns: Scheduling (multiflexi-scheduler), job management (Job class), and execution (multiflexi-executor) are separated into distinct services

  3. Database-Driven Scheduling: The schedule table acts as a persistent queue, ensuring no jobs are lost even if services restart

  4. Flexible Execution: Multiple executor types allow jobs to run in different environments based on requirements

  5. Environment Isolation: Each job has its own serialized environment, preventing interference between executions

  6. Comprehensive Monitoring: Integration with Zabbix and OpenTelemetry provides visibility into the entire lifecycle

  7. Incremental Scheduling: Jobs are scheduled one run at a time (next_schedule is set after execution), preventing duplicate scheduling

  8. Resource Management: Memory limits and parallel execution controls prevent resource exhaustion

Credential Management Architecture

MultiFlexi implements a sophisticated three-tier credential management system that provides secure, reusable, and flexible authentication handling across the entire platform. This architecture ensures clear separation between credential templates, company-specific implementations, and actual usage.

Three-Tier Credential System

The credential management follows a hierarchical relationship:

CredentialPrototype → CredentialType → Credential

┌─────────────────────┐
│ CredentialPrototype │ (JSON Templates - Global)
│                     │
│ • JSON schema       │
│ • Field definitions │
│ • Validation rules  │
│ • Reusable across   │
│   companies         │
└──────────┬──────────┘
           │ instantiated as
           ▼
┌─────────────────────┐
│   CredentialType    │ (PHP Classes - Company Scoped)
│                     │
│ • PHP implementation│
│ • Company-specific  │
│ • Business logic    │
│ • UUID tracking     │
└──────────┬──────────┘
           │ provides schema for
           ▼
┌─────────────────────┐
│     Credential      │ (Values - Application Usage)
│                     │
│ • Actual values     │
│ • Encrypted storage │
│ • Used by jobs      │
│ • RunTemplate refs  │
└─────────────────────┘

Database Schema Relationships:

credential_prototype (1) ←→ (∞) credential_prototype_field
        ↓ (1:∞)
credential_type ←→ (1:1) company
        ↓ (1:∞)
credentials ←→ (∞:1) runtemplate
           ←→ (∞:1) application
Tier 1: CredentialPrototype (JSON Templates)

  • Purpose: Define reusable credential schemas and field structures

  • Storage: credential_prototype and credential_prototype_field tables

  • Format: JSON-based templates with field definitions, types, validation

  • Scope: Global templates available to all companies

  • Management: multiflexi-cli crprototype commands

Tier 2: CredentialType (Company Instances)

  • Purpose: Company-specific implementations of credential prototypes

  • Storage: credential_type table with UUID support

  • Implementation: PHP classes implementing CredentialTypeInterface

  • Scope: Company-isolated instances

  • Management: multiflexi-cli credtype commands

Tier 3: Credential (Application Usage)

  • Purpose: Actual credential values used by applications and jobs

  • Storage: credentials table with encrypted sensitive data

  • Usage: Referenced by RunTemplates and job executions

  • Scope: Specific credential instances for operational use

  • Management: Web interface and CLI tools

Security and Isolation Features

Multi-Tenant Security

  • Company-scoped credential types prevent cross-tenant access

  • Database-level isolation enforced through foreign key constraints

  • Role-based access controls limit credential management permissions

Encryption and Protection

  • Sensitive fields (passwords, API keys) encrypted at rest

  • Separate encryption key management from credential storage

  • Secure credential field transmission between components

Audit and Compliance

  • Complete audit trail for all credential operations

  • GDPR-compliant data handling and deletion capabilities

  • Secure credential rotation and lifecycle management

Template Reusability and Standardization

  • Standardized field definitions across companies using same systems

  • Consistent validation rules and data types

  • Reduced setup complexity for common integration patterns