# GFTS Pre/Post-Hook Design

**Status:** design, ready to implement.
**Extends:** `slot_pool_eval` (5-slot pool pattern, validated).
**Owner:** GFTS-core team.

Adds two capabilities to the transfer subsystem:

1. **Customer script hooks** — arbitrary customer-provided scripts (ICAP scan, DLP, Slack notify, PII redaction, format conversion, etc.) that run inline between transfer legs.
2. **Logical Endpoints (LE)** — MFT-E Virtual Folders as first-class source/destination endpoints, alongside Physical Endpoints (PE).

## What you should already know

- `slot_pool_eval` proves the 5-slot pool works. Pre-deployed `n1..n5` templates handle any batch size via `Ceil(N/5)` parallel `/run/order` calls. See `slot_pool_eval/out/RESULTS1.md` and `RESULTS2.md`.
- The 5-entry `FileTransfers[]` cap is a hard Control-M EM limit (`Validation2034`). See `slot_pool_max/README.md`.
- `ContinueOnFailure=true` on each `FileTransfers[]` entry gives per-file resilience: entry 3 fails → 4, 5 still run. Validated.

**Naming convention.** All template names in this doc use the **production** `gfts_ft_*` prefix. The existing POC templates (`slot_pool_eval/`, `slot_pool_max/`) use a `gk_ft_*` prefix, and any future prototypes under `slot_pool_hooked/` should keep the `gk_ft_*` prefix during development. The rename to `gfts_ft_*` happens when the templates ship in the production bootstrap script (checklist item #1).

## MFT-E REST API constraints (v9.0.22)

Verified against the MFT-E hub source (`ProxyController.java` at `/proxy/*` and `/internal/*`):


| Op       | Single-file                                   | Bulk endpoint                                                                           |
| -------- | --------------------------------------------- | --------------------------------------------------------------------------------------- |
| Upload   | `POST /proxy/files/{folder}`                  | none — one file per call                                                                |
| Download | `GET /proxy/files/{folder}?path=…`            | `**POST /proxy/files/downloadMultipleByZipFile**` — body `String[]`, returns zip        |
| Move     | `PUT /proxy/files/move/{folder}/{destFolder}` | none — one file per call                                                                |
| Delete   | `DELETE /proxy/files/{folder}?path=…`         | `**POST /proxy/files/deleteMultipleFiles/{folder}**` — body `String[]`, per-file status |


Design implication: batching wins in the pool pattern **hold** for PE-leg MFT transfers, Download (zip), and Delete. They **do not hold** for Upload and Move — each is 1 API call per file. We still batch them at the Control-M level (one `Job:AppIntegration` loops through the batch's files sequentially), which saves job dispatch.

## Terminology

- **PE** — Physical Endpoint. SFTP/FTP/local FS reachable directly by a Control-M MFT agent via a connection profile.
- **LE** — Logical Endpoint. An MFT-E Virtual Folder. Agents cannot access it directly; every interaction is a REST API call to MFT-E.
- **Hook** — customer-provided script that runs on the GFTS agent's staging directory between transfer legs.
- **Inspect hook** — script reads the file and decides accept/reject per file. Does not modify bytes.
- **Transform hook** — script rewrites the file (redaction, format convert, encrypt). Modifies bytes.
- **Chain** — an ordered sequence of Control-M jobs that make up one logical transfer, wired via event conditions.
- **Sandwich** — canonical hook chain: `xfer-in → hook → xfer-out`.

## Transfer matrix

Nine families total. Naming pattern: `gfts_ft_<family>_n<k>`, `k ∈ {1..5}`.


| Family               | Chain steps                                    | CM jobs | MFT-E API calls per batch of 5 |
| -------------------- | ---------------------------------------------- | ------- | ------------------------------ |
| `p2p_bare`           | `MFT`                                          | 1       | 0                              |
| `p2p_hook`           | `MFT-in → hook → MFT-out`                      | 3       | 0                              |
| `p2l_bare`           | `MFT-in → Upload×5`                            | 2       | 5                              |
| `p2l_hook`           | `MFT-in → hook → Upload×5`                     | 3       | 5                              |
| `l2l_bare`           | `Move×5`                                       | 1       | 5                              |
| `l2l_hook_inspect`   | `Download-zip → hook → Move×5 → Delete-bulk`   | 4       | 7                              |
| `l2l_hook_transform` | `Download-zip → hook → Upload×5 → Delete-bulk` | 4       | 7                              |
| `l2p_bare`           | `Download-zip → MFT-out → Delete-bulk`         | 3       | 2                              |
| `l2p_hook`           | `Download-zip → hook → MFT-out → Delete-bulk`  | 4       | 2                              |


**Endpoint-to-VD conventions (per user's optimization principle: skip VDs where no useful step happens on them):**

- `p2l`: agent already holds the file after MFT-in, so upload goes **directly** to `dest-org VD`. `src-org VD` is skipped.
- `l2l`: file lives in `src-org VD` from a prior send. Move (or Upload+Delete for transform) delivers to `dest-org VD`.
- `l2p`: file lives in `src-org VD`. Download to staging, ship to dest PE, delete `src-org VD`. `dest-org VD` is never involved.
- Chain shapes above are optimal — no wasted VD hops.

## Pool inventory (pre-deployed at GFTS bootstrap, once per environment)

The templates below get deployed by a GFTS bootstrap script analogous to `slot_pool_eval/1.0_deploy_templates.py`. They live in the same Control-M folder as today's `gfts_ft` templates.

```
# ── BARE pools (no customer hook, fast-path) ──────────────────────────
gfts_ft_p2p_bare_n{1..5}        #  5 templates — PE → PE, plain MFT
gfts_ft_p2l_bare_n{1..5}        #  5 templates — PE → LE, MFT-in + Upload×N
gfts_ft_l2l_bare                #  1 template  — LE → LE, Move-only loop
gfts_ft_l2p_bare_n{1..5}        #  5 templates — LE → PE, Download-zip + MFT-out + Delete
                                # ─────────────────────────────
                                # 16 bare templates

# ── HOOKED pools (customer script mid-chain) ──────────────────────────
gfts_ft_p2p_hook_n{1..5}                #  5 templates — MFT-in + hook + MFT-out
gfts_ft_p2l_hook_n{1..5}                #  5 templates — MFT-in + hook + Upload×N
gfts_ft_l2l_hook_inspect_n{1..5}        #  5 templates — Download-zip + hook + Move×N + Delete
gfts_ft_l2l_hook_transform_n{1..5}      #  5 templates — Download-zip + hook + Upload×N + Delete
gfts_ft_l2p_hook_n{1..5}                #  5 templates — Download-zip + hook + MFT-out + Delete
                                        # ─────────────────────────────
                                        # 25 hooked templates

# Grand total: 41 templates, all deployed idempotently at bootstrap.
```

`l2l_bare` needs only one template because there is no batch-side transfer job — a single `Job:AppIntegration` step contains the Move loop.

`gfts_ft_p2p_bare_n{1..5}` is a rename of today's existing POC templates (`gk_ft_n{1..5}` in `slot_pool_eval/`) — same job shape, production name.

## Chain construction

Every chain is a single Control-M folder ordered as one atomic unit (`ordering=UserDaily` or equivalent). Steps within the folder are wired with event conditions using `%%ORDERID` for concurrency isolation.

### Job types per step


| Step                                      | CM job type          | Notes                                           |
| ----------------------------------------- | -------------------- | ----------------------------------------------- |
| `MFT-in` / `MFT-out`                      | `Job:FileTransfer`   | 5-entry template as today                       |
| `hook`                                    | `Job:Command`        | invokes the customer script wrapper (see below) |
| `Upload` / `Download` / `Move` / `Delete` | `Job:AppIntegration` | uses existing GFTS AI plugin for MFT-E REST     |


### Event wiring pattern

```
xfer_in    :  addEvent  "chain-<%%ORDERID>-xfer-in-done"
hook       :  waitFor   "chain-<%%ORDERID>-xfer-in-done"
              addEvent  "chain-<%%ORDERID>-hook-done"
xfer_out   :  waitFor   "chain-<%%ORDERID>-hook-done"
              addEvent  "chain-<%%ORDERID>-xfer-out-done"
```

`%%ORDERID` is auto-substituted per order; concurrent chains from the same template do not collide on events. Same pattern for 4-job chains (l2l, l2p) — just more `waitFor`/`addEvent` pairs.

### Multi-file API loops (Upload / Move)

Where the MFT-E API is single-file only, the `Job:AppIntegration` step embeds a small in-job loop (e.g. via the AI plugin's pre-execute script, or a wrapper `Job:Command` that calls the AI plugin N times). Implementation choice for the dev: whatever fits the existing AI plugin idiom best.

## Customer script contract

Everything customer-facing lives at `/opt/gfts/customer_scripts/<tenant>/<hook_name>[.sh|.py|...]`.

### Invocation

The hook wrapper (a GFTS-owned shim, not the customer's code) is invoked by the `Job:Command` step with fixed arguments:

```
/opt/gfts/bin/run_hook.sh <hook_name> <manifest_in_path> <manifest_out_path>
```

The shim:

1. Resolves `<hook_name>` to the customer script under the tenant's folder.
2. Enforces `timeout`, resource limits (`systemd-run` or `ulimit`).
3. Executes: `<customer_script> <manifest_in> <manifest_out>`.
4. Captures stdout/stderr to `/opt/gfts/logs/hooks/<transfer_id>/<hook_name>.log`.
5. Returns the customer script's exit code as-is.

### `manifest_in.json` (GFTS writes, customer reads)

```json
{
  "transfer_id":  "%%TRANSFER_ID",
  "order_id":     "%%ORDERID",
  "tenant":       "acme_corp",
  "hook_mode":    "inspect|transform",
  "staging_dir":  "/tmp/gfts/stage/<transfer_id>/",
  "files": [
    {"name": "invoice_001.pdf", "path": "/tmp/gfts/stage/.../invoice_001.pdf"},
    {"name": "invoice_002.pdf", "path": "/tmp/gfts/stage/.../invoice_002.pdf"}
  ],
  "src_endpoint":  {"kind": "PE", "id": "acme_sftp_prod"},
  "dst_endpoint":  {"kind": "LE", "id": "acme_inbox_vd"}
}
```

### `manifest_out.json` (customer writes, GFTS reads)

```json
{
  "results": [
    {"name": "invoice_001.pdf", "status": "ok"},
    {"name": "invoice_002.pdf", "status": "rejected", "reason": "virus_detected"}
  ]
}
```

### Rejection mechanic

The customer script implements rejection by **physically deleting the file from `staging_dir`**. GFTS's downstream step reads `manifest_out.json` for audit but drives file-level control via what's actually present on disk.

For MFT-out / Upload with `ContinueOnFailure=true`, a missing source file marks that entry as failed and moves on — same semantics as `slot_pool_eval` per-file resilience.

For Move / Delete chains, the downstream step re-lists staging and only operates on surviving files.

### Exit codes


| Script exit | Meaning                                                                   | GFTS action                                                     |
| ----------- | ------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `0`         | Script ran cleanly (some files may be rejected via manifest / disk state) | Continue chain                                                  |
| `non-zero`  | Script itself errored (config, unreachable dependency, timeout)           | Fail chain, mark order NOT OK, no retry unless customer opts in |


## Hook registration

Customers register hooks via a GFTS admin API (design out of scope for this doc — separate PR). A registration record contains:

```yaml
tenant:      acme_corp
name:        icap-scan
mode:        inspect              # inspect | transform
script_path: /opt/gfts/customer_scripts/acme_corp/icap-scan.sh
timeout_s:   30
resource_limits:
  memory_mb: 512
  cpu_ms:    5000
```

**Mode is declared at registration and immutable.** If a customer needs both semantics for the same tool, they register two hooks (`icap-scan-only`, `icap-scan-and-quarantine`).

## GFTS-core routing

Given a transfer request, GFTS-core selects the pool + variables:

```python
def pool_for(src_kind, dst_kind, hook_name, batch_size):
    if hook_name is None:
        family = f"{src_kind}2{dst_kind}_bare"
    else:
        hook = registry.get(hook_name)
        if (src_kind, dst_kind) == ("l", "l"):
            family = f"l2l_hook_{hook.mode}"       # inspect | transform
        else:
            family = f"{src_kind}2{dst_kind}_hook"
    slot = f"n{batch_size}" if family != "l2l_bare" else ""
    return f"gfts_ft_{family}_{slot}".rstrip("_")
```

Variables passed at `/run/order`:


| Variable                              | Set for         | Value                                   |
| ------------------------------------- | --------------- | --------------------------------------- |
| `%%TRANSFER_ID`                       | all             | GFTS-issued UUID                        |
| `%%TENANT`                            | all             | for hook resolution and logging         |
| `%%FT_SRC_CP` / `%%FT_DST_CP`         | PE-leg families | connection profile names                |
| `%%FT_SRC_PATH_i` / `%%FT_DST_PATH_i` | PE-leg families | `i ∈ 1..5`                              |
| `%%FT_PATTERN_i`                      | PE-leg families | `Exact` | `Wildcard`                    |
| `%%LE_SRC_VD` / `%%LE_DST_VD`         | LE-leg families | MFT-E folder IDs                        |
| `%%LE_FILE_i`                         | LE-leg families | file names within the VD, `i ∈ 1..5`    |
| `%%HOOK_NAME`                         | hook families   | resolves to `<script_path>` in the shim |
| `%%STAGING_DIR`                       | all             | `/tmp/gfts/stage/%%TRANSFER_ID/`        |


## Failure semantics

Per-family behavior when a specific step fails. `CONT_EXE` still governs within-step per-file behavior; the table below is *cross-step*.


| Failed step                   | Behavior                                                                                    |
| ----------------------------- | ------------------------------------------------------------------------------------------- |
| `MFT-in`                      | Chain fails. No staging cleanup needed for MFT-in itself (nothing landed). Order NOT OK.    |
| `hook` (exit non-zero)        | Chain fails. Staging cleanup runs on retry / manual intervention. Order NOT OK.             |
| `Upload` (per-file, mid-loop) | Continue to next file (loop's own `on_failure=continue`). Order NOT OK if any file failed.  |
| `Move` (per-file, mid-loop)   | Continue to next file. Order NOT OK if any failed.                                          |
| `MFT-out`                     | `ContinueOnFailure=true` at file level. Order NOT OK if any entry failed.                   |
| `Delete` (bulk, cleanup)      | Log and continue. Never fail the chain over cleanup. Order stays OK if only cleanup failed. |


Retry: use Control-M's `RerunSpecificationSet` at the folder level. Idempotency requirements: the customer script must be safe to re-invoke with the same manifest. GFTS's own steps (Upload/Move/Delete/MFT) are effectively idempotent when combined with `ContinueOnFailure` and staging cleanup.

## State propagation between chain steps

State passed via `Variables[]` at order time and via files at `%%STAGING_DIR`:

- `%%TRANSFER_ID` and `%%ORDERID` — variable, all steps.
- `%%STAGING_DIR` — variable, all steps that touch staging.
- `manifest_in.json` / `manifest_out.json` — files under `%%STAGING_DIR/`, live for the chain's lifetime.
- File content — under `%%STAGING_DIR/`, deleted by the final cleanup step (or by the customer script on rejection).

No external state store required for the chain itself. Audit/tracking data still flows to GFTS's normal `transfer_run` tables via the AI plugin's post-execute hook.

## What's out of scope for this doc

Deliberately deferred to keep the initial implementation tight:

1. **Hook registration API** — separate design.
2. **Multi-tenant isolation hardening** — Unix user separation, egress proxy, per-tenant `systemd` slices. Documented as future work in `docs/security/hook-sandbox.md`.
3. **Post-final hooks** — hooks after the last transfer leg (e.g., "notify after delivery"). Current design only covers mid-chain hooks. Adding a post-final hook is a straightforward extension of the chain — reserve the naming but don't build.
4. **Cross-tenant script sharing** — customers cannot share hooks across tenants in v1.
5. **Per-file parallelism inside a batch's Upload/Move loop** — v1 does these sequentially. Parallelism is a v2 optimization if numbers demand it.

## Implementation checklist

Roughly in order. Each item is independently testable.

- [ ] **1. Rename existing pool.** POC templates `gk_ft_n{1..5}` become production `gfts_ft_p2p_bare_n{1..5}`. Existing `slot_pool_eval/` scripts stay on the `gk_ft_*` names and are untouched; the bootstrap deploys the `gfts_ft_*` templates alongside so both can coexist during rollout.
- [ ] **2. Build `run_hook.sh` shim.** Wraps customer script, enforces timeout + resource limits, captures logs, writes/reads manifests.
- [ ] **3. Build `p2p_hook_n{1..5}` templates.** Sandwich chain: `MFT-in → Job:Command(run_hook.sh) → MFT-out`. Event wiring with `%%ORDERID`.
- [ ] **4. Wire GFTS-core routing for `p2p_bare` and `p2p_hook`.** Extend the dispatcher to select family + set variables. Add `hook_name` to the transfer request DTO. Feature-flag it OFF until validated.
- [ ] **5. Prototype an ICAP-scan hook and a NOOP hook.** Register them, run p2p_hook end-to-end with both, verify:
  - happy path,
  - hook rejects one file (per-file deletion + `ContinueOnFailure` behavior),
  - hook errors out (chain fails, staging preserved for debug).
- [ ] **6. Measure sandwich tax.** Compare `p2p_bare` vs `p2p_hook` wall-clock for `N=100`. Expect 2–3× because 3 jobs per chain vs 1. Document in a new `slot_pool_hooked/RESULTS1.md`.
- [ ] **7. Build the AI-plugin `Upload` / `Download` / `Move` / `Delete` job templates.** Use existing GFTS AI plugin. Verify against a real MFT-E instance.
- [ ] **8. Build `p2l_`* and `l2p_*` families.** Same chain construction pattern; different job types.
- [ ] **9. Build `l2l_bare` (single Move loop).** Simplest LE family; validates the Move-loop pattern.
- [ ] **10. Build `l2l_hook_inspect` and `l2l_hook_transform`.** Last, most complex.
- [ ] **11. Bootstrap script.** Deploy all 41 templates idempotently at GFTS install/upgrade.

## Open questions for the developer to decide during implementation

1. **AI-plugin loop location** — implement multi-file Upload/Move as a loop inside the AI plugin's `pre-execute` script, or as multiple `Job:AppIntegration` invocations chained by events? Pick whichever fits the existing plugin idiom.
2. **Zip unpack step** — after Download-zip, add an explicit `Job:Command` to unzip into staging, or embed the unzip in the AI plugin's `post-execute`? Same trade-off.
3. **Timeout enforcement in `run_hook.sh`** — `timeout` GNU util is simplest; `systemd-run --scope --property=RuntimeMaxSec=` is stricter. Pick based on host OS.

## Cross-reference

- Batching semantics + `ContinueOnFailure`: `../slot_pool_eval/README.md`
- 5-entry hard cap evidence: `../slot_pool_max/README.md`
- MFT-E API source spelunking: this doc's "MFT-E REST API constraints" section, backed by `mft-hub/src/main/java/com/bmc/ctm/mft/b2b/hub/server/controller/ProxyController.java` in AFT 9.0.22.