> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ionworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026 04 03 ucp simulation backend design optimization

# UCP Simulation Backend for Design Optimization

**Date:** 2026-04-03
**Status:** Draft

## Problem

Design optimization currently converts UCP protocol YAML to a PyBaMM `Experiment` object
via `ucp_to_pybamm_experiment()`, then runs it through `pybamm.Simulation.solve()`. This
conversion is lossy (no dynamic loops, conditionals, goto, EIS handling), fragile (recent
bugs around expression parsing and cycle structure), and slower than necessary (no model
caching between steps).

## Solution

Add a `UCPSimulation` adapter class that implements the same `.solve(inputs=...)` interface
as `iwp.Simulation`, but internally uses UCP's native `build_protocol_simulation()` /
`run_protocol_simulation()`. The adapter returns `ucp_solution.timeseries_solution` — a
real `pybamm.Solution` built from internal state vectors — so all existing metrics,
actions, constraints, and cycle slicing work unchanged.

Additionally, block EIS steps in design optimization experiments since
`timeseries_solution` excludes them and they are not meaningful in optimization contexts.

## Architecture

### UCPSimulation Adapter

A new class in `ionworkspipeline` that duck-types `iwp.Simulation` for the subset of the
interface that `DesignObjective` uses:

```python theme={null}
class UCPSimulation:
    """Simulation adapter using UCP's native protocol runner.

    Provides the same .solve(inputs=...) interface as iwp.Simulation but runs the
    protocol through UCP's step-by-step engine, which supports dynamic loops,
    conditionals, input parameter expressions, and aggressive model caching.

    Returns ucp_solution.timeseries_solution — a real pybamm.Solution built from
    internal state vectors — so all existing metrics work unchanged.
    """

    def __init__(self, model, parameter_values, protocol_yaml, solver=None, **kwargs):
        # Parse UCP protocol from YAML string
        # Build simulation environment once (model discretization, caching)
        self._sim_env = build_protocol_simulation(
            protocol=protocol_yaml,
            model=model,
            parameter_values=parameter_values,
            **kwargs,
        )

    def solve(self, inputs=None, initial_soc=None, save_at_cycles=None, **kwargs):
        # Run protocol step-by-step through UCP engine
        ucp_solution = run_protocol_simulation(self._sim_env, inputs=inputs)
        # Build pybamm.Solution with cycle structure from UCP step metadata
        return self._build_solution_with_cycles(ucp_solution)
```

### Cycle Structure on the Returned Solution

`UCPSolution.timeseries_solution` builds a flat `pybamm.Solution` with `.sub_solutions`
(one per timeseries step) but `.cycles` is empty — PyBaMM only populates cycles when
solved via `pybamm.Simulation` with a `pybamm.Experiment`.

`UCPSimulation.solve()` fixes this by building cycle boundaries from UCP's step metadata.
Each `StepSolution` already has a `.cycle_count` field. The adapter groups sub-solutions
by cycle count and constructs a `pybamm.Solution` per cycle:

```python theme={null}
def _build_solution_with_cycles(self, ucp_solution):
    pybamm_sol = ucp_solution.timeseries_solution

    # Group timeseries step indices by cycle_count
    cycle_groups = {}  # cycle_count -> [step_indices]
    step_idx = 0
    for step_sol in ucp_solution.steps:
        if not step_sol.is_timeseries:
            continue
        cycle_groups.setdefault(step_sol.cycle_count, []).append(step_idx)
        step_idx += 1

    # Build a pybamm.Solution per cycle from subsets of the flat arrays
    cycles = []
    for cycle_count in sorted(cycle_groups):
        indices = cycle_groups[cycle_count]
        cycle_sol = pybamm.Solution(
            all_ts=[pybamm_sol.all_ts[i] for i in indices],
            all_ys=[pybamm_sol.all_ys[i] for i in indices],
            all_yps=[pybamm_sol.all_yps[i] for i in indices],
            all_models=[pybamm_sol.all_models[i] for i in indices],
            all_inputs=[pybamm_sol.all_inputs[i] for i in indices],
            termination="final time",
        )
        cycles.append(cycle_sol)

    pybamm_sol.cycles = cycles
    return pybamm_sol
```

This gives full cycle support:

* `sol.cycles[c]` — real `pybamm.Solution` with full interpolation
* `sol.cycles[c].sub_solutions` — steps within that cycle
* `save_at_cycles` — works as expected
* `CycleSlicer`, `IterativeMetric` — work unchanged

**Key details:**

* `initial_soc`: UCP protocols encode initial state in their `global` config
  (`initial_state_type` / `initial_state_value`). When using the UCP backend,
  `parse_objectives()` must NOT extract initial SOC from the experiment config and inject it
  into `parameters` (which it currently does at lines 145-160). Otherwise initial SOC is
  applied twice — once by `parse_objectives` into `parameter_values` and again by
  `run_protocol_simulation` from the protocol's `global_variables`. The fix: skip the
  initial-state extraction in `parse_objectives()` when `backend="ucp"`, letting UCP handle
  it exclusively from the protocol YAML.
* `build_protocol_simulation` is called once in `build()`. `run_protocol_simulation` is
  called on every optimizer iteration in `_run()`. UCP's `FastPybammSimulation` caches
  step-specific built models across iterations.
* **Solver kwargs**: `build_protocol_simulation` accepts `**kwargs` that are passed to the
  solver. `_set_up_ucp_simulation` should extract solver-related kwargs from
  `simulation_kwargs` (e.g., tolerances) and forward them, matching the existing behavior of
  `set_up_simulation` in `BaseObjective`.

### Known Limitation: UCP-Computed Columns

UCP's DataFrame output includes computed columns (`Charge capacity [A.h]`,
`Charge energy [W.h]`) that are calculated via trapezoidal integration of current/power.
These are **not available** through `timeseries_solution` because they are not PyBaMM state
variables.

However, PyBaMM's own state variables — `Discharge capacity [A.h]`,
`Discharge energy [W.h]`, `Throughput capacity [A.h]` — are in the `y` state vector and
fully available via `sol["Discharge capacity [A.h]"]`. These are the variables that
ionworkspipeline metrics reference. The UCP-specific charge/discharge split columns are a
DataFrame convenience not used by the optimization system.

### Backend Selection in DesignObjective

In `DesignObjective.build()`, select the simulation backend based on an `options["backend"]`
field, with fallback to the `IONWORKS_SIMULATION_BACKEND` environment variable:

```python theme={null}
def build(self, parameter_values):
    # ... existing variable processing (unchanged) ...

    backend = self.options.get("backend") or os.environ.get(
        "IONWORKS_SIMULATION_BACKEND", "pybamm"
    )

    if backend == "ucp":
        self.simulation = self._set_up_ucp_simulation(model, parameter_values)
    else:
        self.simulation = self.set_up_simulation(model, parameter_values)
```

`_set_up_ucp_simulation` extracts the experiment YAML string from
`self.options["simulation_kwargs"]["experiment"]` (before it gets parsed to a PyBaMM
Experiment), creates a `UCPSimulation`, and returns it.

**Important:** When `backend="ucp"`, the experiment string must be preserved as a raw YAML
string and NOT converted to a `pybamm.Experiment` by `parse_experiment()`. The approach:
in `parse_options()` (objectives.py line \~298), check if `options_config.get("backend") ==
"ucp"` before the `parse_experiment()` call. If so, skip the conversion and leave the
experiment as a raw string in `simulation_kwargs["experiment"]`. This is a small, contained
change — `parse_options()` already inspects `simulation_kwargs` contents, so adding a
backend check is natural. The raw string is then available for `_set_up_ucp_simulation` to
pass directly to `UCPSimulation`.

**`_validate_solution_steps` handling:** `DesignObjective._validate_solution_steps()` calls
`self._get_experiment_steps()` which accesses `experiment.steps` — this fails on a raw YAML
string. When `backend="ucp"`, set `self._validate_against_experiment_steps = False` in
`build()` before creating the simulation. UCP's own protocol validation (step count, step
types) is sufficient — the PyBaMM-specific step count check is not meaningful for UCP runs.

**`to_config()` round-trip:** `DesignObjective.to_config()` checks
`hasattr(experiment, "steps")` — a raw YAML string doesn't have `.steps`, so the string
passes through as-is. This is correct behavior. The `backend` field in `options` also
survives serialization since `options` is a plain dict. Checkpoint/restart of UCP-backed
objectives works without additional changes.

### Config Passthrough

The `backend` option flows through the existing config path with no schema changes:

1. **API request**: Add optional `backend` field to the optimization request (or use env
   var only — no API change needed for MVP).
2. **Backend converter** (`design_optimization_converter.py`): If a `backend` field is
   present in the request, include `"backend": "ucp"` in each objective's `options` dict.
3. **`parse_objectives()`**: Passes `options` through to `DesignObjective.__init__()` — no
   changes needed since `options` is `dict[str, Any]`.
4. **`DesignObjective.build()`**: Reads `self.options["backend"]` as described above.

### EIS Validation

EIS steps are not supported in design optimization. Validation happens at two levels:

**Frontend (blocks submit):**

In `handleCreateOptimization` in `optimization-new-with-template-container.tsx`, check each
objective's experiment string with a simple regex before submission:

```typescript theme={null}
const hasEisSteps = (experiment: string) => /\beis\b/i.test(experiment);

for (const obj of objectives) {
    if (hasEisSteps(obj.experiment)) {
        toast.error('EIS steps are not supported in design optimization');
        return;
    }
}
```

**Backend (defense-in-depth):**

In `convert_to_datafit_job_params()` in `design_optimization_converter.py`, apply the same
check before job submission:

```python theme={null}
import re

def _validate_no_eis_steps(experiment_yaml: str) -> None:
    if re.search(r"\beis\b", experiment_yaml, re.IGNORECASE):
        raise BadRequestError(
            "EIS steps are not supported in design optimization experiments"
        )
```

This gives an immediate 400 error rather than a failed job.

## What Changes

| Component                                      | Change                                            |
| ---------------------------------------------- | ------------------------------------------------- |
| `ionworkspipeline` — new `UCPSimulation` class | New file or added to existing simulation module   |
| `DesignObjective.build()`                      | Add backend selection logic                       |
| `parse_objectives()`                           | Skip `parse_experiment()` when backend is `"ucp"` |
| `design_optimization_converter.py`             | Pass through `backend` option, validate no EIS    |
| Frontend optimization form                     | Add EIS regex check before submit                 |

## What Doesn't Change

* `DesignObjective._run()` — unchanged, calls `self.simulation.solve(inputs=...)`
* All metrics and actions — unchanged, receive `pybamm.Solution`
* `DataFitJobProcessor` — unchanged
* `DataFit` optimizer loop — unchanged
* Frontend form structure — unchanged (just one validation added)

## Testing

1. **Unit: `UCPSimulation.solve()`** — returns valid `pybamm.Solution` with expected
   variables, `.sub_solutions`, and `.cycles`
2. **Unit: cycle structure** — `.cycles` correctly groups steps by `StepSolution.cycle_count`;
   each cycle has its own `.sub_solutions`; `save_at_cycles` works
3. **Unit: EIS validation** — frontend regex and backend check both reject protocols with
   EIS steps, accept protocols without
4. **Unit: backend selection** — `DesignObjective.build()` creates `UCPSimulation` when
   `backend="ucp"`, `iwp.Simulation` when `backend="pybamm"`
5. **Unit: env var fallback** — `IONWORKS_SIMULATION_BACKEND=ucp` selects UCP when no
   explicit option is set
6. **Unit: `_validate_solution_steps` disabled** — confirm it is set to `False` for UCP
   backend and does not interfere with simulation results
7. **Unit: `to_config()` round-trip** — serialize and deserialize a UCP-backed objective,
   verify `backend` field and raw experiment YAML survive
8. **Unit: initial SOC not double-applied** — verify `parse_objectives()` skips initial
   state extraction when `backend="ucp"`
9. **Integration: end-to-end optimization** — run a design optimization with
   `backend="ucp"` and verify it produces equivalent results to `backend="pybamm"`
10. **Regression** — existing `backend="pybamm"` path works identically

## Risks and Mitigations

* **`timeseries_solution` fidelity**: The `pybamm.Solution` built from UCP state vectors
  should be equivalent to one from `pybamm.Simulation.solve()`. This is already tested in
  the UCP package. Integration tests will verify metric values match between backends.
* **Experiment string preservation**: Addressed by checking `backend="ucp"` in
  `parse_options()` to skip `parse_experiment()`. Small, contained change.
* **Initial SOC double application**: Addressed by skipping initial-state extraction in
  `parse_objectives()` when `backend="ucp"`. UCP handles it from the protocol YAML.
* **Cycle structure**: `UCPSimulation.solve()` builds `.cycles` by grouping sub-solutions
  using `StepSolution.cycle_count`. Each cycle is a real `pybamm.Solution` with full
  interpolation. `save_at_cycles`, `CycleSlicer`, and `IterativeMetric` all work.
* **UCP-computed columns**: `Charge capacity [A.h]` and `Charge energy [W.h]` (UCP
  DataFrame columns) are not available via `timeseries_solution`. PyBaMM's own
  `Discharge capacity [A.h]` and `Discharge energy [W.h]` state variables are available and
  are what ionworkspipeline metrics use.
* **`_validate_solution_steps` incompatibility**: Disabled for UCP backend. UCP's own
  protocol validation is sufficient.
* **Converter uses `HTTPException`**: The existing `design_optimization_converter.py` raises
  `HTTPException` directly, violating the project's layered architecture rules. The new EIS
  validation uses `BadRequestError` correctly. Cleaning up existing `HTTPException` usage is
  out of scope but noted as tech debt.