Diagnostics and post-mortem

What to do when production is already broken: a generic triage order, then the X3 V12 screens that answer “who is connected”, “who holds the lock”, “why did the batch fail” and “who changed this row”, plus a database-side checklist and an incident report template. Tracing techniques themselves are in debugging-traces.md; slowness analysis in performance.md.

Contents

Triage order

Generic, tool-independent order for the first minutes of a P1:

  1. Scope — one user, one function, one site, or everybody? One endpoint (folder) or all of them?
  2. Layer — browser/Syracuse (pages do not load at all), X3 runtime (functions fail, sessions die), database (everything waits), or business logic (one rule rejects)?
  3. Change — last patch, last activity-code or parameter change, last deployment of specific code, last batch.
  4. Capture before you fix — session list, lock list, request log, DB blocking chain, screenshots. Restarting a service or killing sessions destroys the evidence.
  5. Contain — workaround (disable a recurring task, ask users to leave a function) before root-causing.
  6. Fix on a copy first — reproduce in a test folder; deliver the fix as a patch, never by editing production.

Symptom to first screen

Symptom Look first at
“Record being modified by another user” style messages on one record Locked symbols (VERSYMB)
Save hangs for several users, no error Database blocking chain (below), then VERSYMB
A batch task shows Error / Warning / stays In progress Request management (ASYRREQMAN) → Log
Everybody slow Sessions information, then DB waits; performance.md
SOAP/REST callers fail X3 session logs (type Web service), integration log — web-services-integration.md
Wrong values in a record Audit trail (AUDITH/AUDITL) or custom audit table — audit-compliance.md
Runtime errors with script/line Log file of the function (TRA), errn/errl traces — debugging-traces.md

Sessions and users

Before disconnecting anything, record user, endpoint, process ids and what the session was running.

Locked symbols vs database locks

Two different mechanisms produce “locked” symptoms:

  Symbol lock Database row lock
Created by Lock SYMBOL (V6-style object management), stored in table APLLCK ([S]adxtlk) Readlock, For … With Lock, any row modified inside a transaction
Lifetime Until Unlock, or until the owning session is ended Until Commit / Rollback
Visible in Locked symbols function VERSYMB DB tools only (blocking query below)
Typical cause Browser closed without logout (“phantom” session, community-reported), crashed process Long transaction, user prompt inside a transaction, runaway batch

VERSYMB (V11 menu Development > Utilities > Verifications > Locks > Locked symbols) lists the symbol, machine, user, X3 identifier (adxuid(1) of the session) and date-time. Object symbols are the object code followed by the key (e.g. AUSMARTIN for user MARTIN; multi-part keys put the second component first, separated by \). The User Monitor action jumps to the session holding the symbol; ending that session releases it. Sage’s Lock documentation notes that V7-style code relies on optimistic locking (Rewritebykey + UPDTICK) instead of symbol locks — see database.md.

Order of preference to clear a lock: let the user finish/leave the record → end the owning session from the monitor → database-level kill by a DBA (rolls back that session’s open transaction). Never delete APLLCK rows by hand without Sage support.

Batch failures

Request management (ASYRREQMAN) lists every request sent to the batch server with folder, task, user, dates, session id, timeout and status (local menu 21): Standby, In progress, Finished, Held, Kill, Canceled, Error, Overdue, Warning (task ended on a non-blocking code, GERRBATCH < 100).

  1. Open the request line → Log: the request trace RQT<request number> from the TRA directory of the runtime’s SERVX3 directory. The server-level Log action shows server.tra (server start, request launch, end).
  2. Open the function’s own log (F<n>.tra or your ALOG file) via LECTRACE/AREADLOG — debugging-traces.md.
  3. If the log is empty, reproduce with an X3 session logs entry of type Batch query (filter on user, task code) or with openlog around the suspect call.
  4. Check Parameter entry on the request: the values the task really received.
  5. After the fix, relaunch once manually, then re-enable the recurring task (GESABA) — batch-scheduling.md.

Since 2025 R1 (V12.0.37) ASYRREQMAN has extra search/sort/filter features and a “Classic function” action giving access to the classic AREQUETE function.

Engine and Syracuse logs

Database-side checks

Generic DBA queries (not X3-specific) to find who blocks whom. Run them while the problem is happening.

-- SQL Server: blocked requests and their blocker
SELECT r.session_id, r.blocking_session_id, r.wait_type, r.wait_time,
       s.host_name, s.program_name, s.host_process_id, t.text
FROM sys.dm_exec_requests r
JOIN sys.dm_exec_sessions s ON s.session_id = r.session_id
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) t
WHERE r.blocking_session_id <> 0;

-- Oracle: blocked sessions and their blocker
SELECT sid, serial#, username, program, machine, blocking_session, event, wait_class
FROM v$session
WHERE blocking_session IS NOT NULL;

Then: follow the chain to the head blocker, map it to an X3 process (host/process id), check what it runs (request management, session list) before any kill. Also check free disk space (TRA, TMP, database logs), DB log/redo saturation and backups running at the same time.

Who changed this data?

Incident report template

INCIDENT:        <one-line summary>
DATE/TIME:       <start - end, timezone>
DETECTED BY:     <user / monitoring / customer>
SCOPE:           <endpoints (folders), functions, users, sites affected>
X3 CONTEXT:      <V12 patch level, last patch applied, activity codes changed>
EVIDENCE:        <request numbers + RQT logs, TRA files, VERSYMB / session screenshots, DB blocking output>
TIMELINE:        <detection, escalation, actions tried with time>
ROOT CAUSE:      <technical, one paragraph>
TRIGGER:         <what made the latent issue surface>
FIX:             <code / patch / parameter / infrastructure change>
DETECTION GAP:   <why it was not caught earlier>
PREVENTION:      <test added (unit-testing-axunit.md), monitoring, review rule>

Gotchas

See also: debugging-traces.md, performance.md, batch-scheduling.md, database.md, audit-compliance.md, web-services-integration.md, version-caveats.md.

Sources