Workers

Worker services are the scan execution engines of S4E On-Prem. They run CPU-intensive tasks and are automatically pinned to dedicated CPU cores to avoid interfering with other services.

s4e-scan

The primary vulnerability scanning worker. It executes Nuclei-based scan templates pulled from S4E Cloud.

Container: s4e-scan-1
CPU pinning: Upper half of available cores (e.g., cores 14–27 on a 28-core server)
Memory limit: 16 GB (configurable via SCAN_MEMORY_LIMIT)

How it works

  1. Receives scan jobs from RabbitMQ (dispatched by s4e-dispatcher).
  2. Loads scan templates from the local volume (populated by s4e-scan-func-puller).
  3. Executes templates against the target asset.
  4. Publishes results back through RabbitMQ.

Key environment variables

Variable Description
SCAN_INFRA Set to onprem for on-premises mode
PYTHONPATH Points to scan template directories
DNS_RESOLVERS DNS servers used for target resolution

Viewing scan logs

docker logs s4e-scan-1 -f

Checking template availability

ls /opt/s4e/volumes/s4e-func/

s4e-crawler

The web crawler worker. It discovers endpoints, parameters, technologies, and API surfaces on web applications.

Container: s4e-crawler-1
CPU pinning: Same as s4e-scan (shares the upper CPU cores)

Crawler pipeline stages

The crawler processes targets through multiple sequential stages:

  1. ffuf — fast web fuzzer, discovers hidden paths and parameters
  2. katana — modern web crawler for JavaScript-heavy apps
  3. api-doc-parser — parses OpenAPI/Swagger docs
  4. url-unifier — deduplicates and normalizes discovered URLs
  5. pii-parser — detects potential PII exposure in responses
  6. enrichment — adds technology fingerprinting
  7. finisher — writes final results to the database

Viewing crawler logs

docker logs s4e-crawler-1 -f

s4e-action

Executes remediation and automation actions defined in playbooks.

Container: s4e-action-1

Actions can include: - Sending notifications (Slack, email, webhook) - Creating tickets (Jira, ServiceNow) - Triggering CI/CD pipelines - Custom scripts

s4e-scan-func-puller

Pulls scan templates from S4E Cloud and stores them locally for s4e-scan to use.

Container: s4e-scan-func-puller-1
Pull interval: Every 6 hours (configurable via SCAN_FUNC_PULL_INTERVAL_SEC)

What it pulls

  • s4e-func — official S4E scan template library
  • s4e-func-custom — your organization's custom templates (if any)

Viewing puller logs

docker logs s4e-scan-func-puller-1 -f

Successful pull: 

[pull-func] SUCCESS: extracted to /opt/s4e/volumes/s4e-func

Auth failure: 

[pull-func] ERROR: 401 Client Error: Unauthorized

If you see auth errors, verify TOKEN and S4E_CLOUD_API_URL in .env.

Resource Tuning

CPU Pinning

Worker CPU pinning is stored in .env as SCAN_CPUSET and CRAWLER_CPUSET. Values are set automatically by setup.sh but can be overridden:

# Pin scan to cores 8-15
SCAN_CPUSET=8-15

# Pin crawler to cores 0-7
CRAWLER_CPUSET=0-7

After changing, restart the affected services:

cd /opt/s4e/setup
docker compose up -d --no-deps scan crawler

Memory Limits

# In .env
SCAN_MEMORY_LIMIT=32G   # increase for large scan volumes