Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Data Flow Overhaul

Browse Source 
Known regressions in current commit:
- No "extra" files
- GLOSS map does not look corrected
- "override" flag is not respected
This commit is contained in:
Rusfort
2025-05-01 09:13:20 +02:00
parent 153e464387
commit 6971b8189f
14 changed files with 3670 additions and 1131 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
ProjectNotes/Data_Flow_Refinement_Plan.md Normal file
View File

@@ -0,0 +1,124 @@
# Architectural Plan: Data Flow Refinement (v3)
**Date:** 2025-04-30
**Author:** Roo (Architect Mode)
**Status:** Approved
## 1. Goal
Refine the application's data flow to establish the GUI as the single source of truth for processing rules. This involves moving prediction/preset logic upstream from the backend processor and ensuring the backend receives a *complete* `SourceRule` object for processing, thereby simplifying the processor itself. This version of the plan involves creating a new processing module (`processing_engine.py`) instead of refactoring the existing `asset_processor.py`.
## 2. Proposed Data Flow
The refined data flow centralizes rule generation and modification within the GUI components before passing a complete, explicit rule set to the backend. The `SourceRule` object structure serves as a consistent data contract throughout the pipeline.
```mermaid
sequenceDiagram
participant User
participant GUI_MainWindow as GUI (main_window.py)
participant GUI_Predictor as Predictor (prediction_handler.py)
participant GUI_UnifiedView as Unified View (unified_view_model.py)
participant Main as main.py
participant ProcessingEngine as New Backend (processing_engine.py)
participant Config as config.py
User->>+GUI_MainWindow: Selects Input & Preset
Note over GUI_MainWindow: Scans input, gets file list
GUI_MainWindow->>+GUI_Predictor: Request Prediction(File List, Preset Name, Input ID)
GUI_Predictor->>+Config: Load Preset Rules & Canonical Types
Config-->>-GUI_Predictor: Return Rules & Types
%% Prediction Logic (Internal to Predictor)
Note over GUI_Predictor: Perform file analysis (based on list), apply preset rules, generate COMPLETE SourceRule hierarchy (only overridable fields populated)
GUI_Predictor-->>-GUI_MainWindow: Return List[SourceRule] (Initial Rules)
GUI_MainWindow->>+GUI_UnifiedView: Populate View(List[SourceRule])
GUI_UnifiedView->>+Config: Read Allowed Asset/File Types for Dropdowns
Config-->>-GUI_UnifiedView: Return Allowed Types
Note over GUI_UnifiedView: Display rules, allow user edits
User->>GUI_UnifiedView: Modifies Rules (Overrides)
GUI_UnifiedView-->>GUI_MainWindow: Update SourceRule Objects in Memory
User->>+GUI_MainWindow: Trigger Processing
GUI_MainWindow->>+Main: Send Final List[SourceRule]
Main->>+ProcessingEngine: Queue Task(SourceRule) for each input
Note over ProcessingEngine: Execute processing based *solely* on the provided SourceRule and static config. No internal prediction/fallback.
ProcessingEngine-->>-Main: Processing Result
Main-->>-GUI_MainWindow: Update Status
GUI_MainWindow-->>User: Show Result/Status
```
## 3. Module-Specific Changes
* **`config.py`:**
* **Add Canonical Lists:** Introduce `ALLOWED_ASSET_TYPES` (e.g., `["Surface", "Model", "Decal", "Atlas", "UtilityMap"]`) and `ALLOWED_FILE_TYPES` (e.g., `["MAP_COL", "MAP_NRM", ..., "MODEL", "EXTRA", "FILE_IGNORE"]`).
* **Purpose:** Single source of truth for GUI dropdowns and validation.
* **Existing Config:** Retains static definitions like `IMAGE_RESOLUTIONS`, `MAP_MERGE_RULES`, `JPG_QUALITY`, etc.
* **`rule_structure.py`:**
* **Remove Enums:** Remove `AssetType` and `ItemType` Enums. Update `AssetRule.asset_type`, `FileRule.item_type_override`, etc., to use string types validated against `config.py` lists.
* **Field Retention:** Keep `FileRule.resolution_override` and `FileRule.channel_merge_instructions` fields for structural consistency, but they will not be populated or used for overrides in this flow.
* **`gui/prediction_handler.py` (or equivalent):**
* **Enhance Prediction Logic:** Modify `run_prediction` method.
* **Input:** Accept `input_source_identifier` (string), `file_list` (List[str] of relative paths), and `preset_name` (string) when called from GUI.
* **Load Config:** Read `ALLOWED_ASSET_TYPES`, `ALLOWED_FILE_TYPES`, and preset rules.
* **Relocate Classification:** Integrate classification/naming logic (previously in `asset_processor.py`) to operate on the provided `file_list`.
* **Generate Complete Rules:** Populate `SourceRule`, `AssetRule`, and `FileRule` objects.
* Set initial values only for *overridable* fields (e.g., `asset_type`, `item_type_override`, `target_asset_name_override`, `supplier_identifier`, `output_format_override`) based on preset rules/defaults.
* Explicitly **do not** populate static config fields like `FileRule.resolution_override` or `FileRule.channel_merge_instructions`.
* **Temporary Files (If needed for non-GUI):** May need logic later to handle direct path inputs (CLI/Docker) involving temporary extraction/cleanup, but the primary GUI flow uses the provided list.
* **Output:** Emit `rule_hierarchy_ready` signal with the `List[SourceRule]`.
* **NEW: `processing_engine.py` (New Module):**
* **Purpose:** Contains a new class (e.g., `ProcessingEngine`) for executing the processing pipeline based solely on a complete `SourceRule` and static configuration. Replaces `asset_processor.py` in the main workflow.
* **Initialization (`__init__`):** Takes the static `Configuration` object as input.
* **Core Method (`process`):** Accepts a single, complete `SourceRule` object. Orchestrates processing steps (workspace setup, extraction, map processing, merging, metadata, organization, cleanup).
* **Helper Methods (Refactored Logic):** Implement simplified versions of processing helpers (e.g., `_process_individual_maps`, `_merge_maps_from_source`, `_generate_metadata_file`, `_organize_output_files`, `_load_and_transform_source`, `_save_image`).
* Retrieve *overridable* parameters directly from the input `SourceRule`.
* Retrieve *static configuration* parameters (resolutions, merge rules) **only** from the stored `Configuration` object.
* Contain **no** prediction, classification, or fallback logic.
* **Dependencies:** `rule_structure.py`, `configuration.py`, `config.py`, cv2, numpy, etc.
* **`asset_processor.py` (Old Module):**
* **Status:** Remains in the codebase **unchanged** for reference.
* **Usage:** No longer called by `main.py` or GUI for standard processing.
* **`gui/main_window.py`:**
* **Scan Input:** Perform initial directory/archive scan to get the file list for each directory/archieve.
* **Initiate Prediction:** Call `PredictionHandler` with the file list, preset, and input identifier.
* **Receive/Pass Rules:** Handle `rule_hierarchy_ready`, pass `SourceRule` list to `UnifiedViewModel`.
* **Send Final Rules:** Send the final `SourceRule` list to `main.py`.
* **`gui/unified_view_model.py` / `gui/delegates.py`:**
* **Load Dropdown Options:** Source dropdowns (`AssetType`, `ItemType`) from `config.py`.
* **Data Handling:** Read/write user modifications to overridable fields in `SourceRule` objects.
* **No UI for Static Config:** Do not provide UI editing for resolution or merge instructions.
* **`main.py`:**
* **Receive Rule List:** Accept `List[SourceRule]` from GUI.
* **Instantiate New Engine:** Import and instantiate the new `ProcessingEngine` from `processing_engine.py`.
* **Queue Tasks:** Iterate `SourceRule` list, queue tasks.
* **Call New Engine:** Pass the individual `SourceRule` object to `ProcessingEngine.process` for each task.
## 4. Rationale / Benefits
* **Single Source of Truth:** GUI holds the final `SourceRule` objects.
* **Backend Simplification:** New `processing_engine.py` is focused solely on execution based on explicit rules and static config.
* **Decoupling:** Reduced coupling between GUI/prediction and backend processing.
* **Clarity:** Clearer data flow and component responsibilities.
* **Maintainability:** Easier maintenance and debugging.
* **Centralized Definitions:** `config.py` centralizes allowed types.
* **Preserves Reference:** Keeps `asset_processor.py` available for comparison.
* **Consistent Data Contract:** `SourceRule` structure is consistent from predictor output to engine input, enabling potential GUI bypass.
## 5. Potential Issues / Considerations
* **`PredictionHandler` Complexity:** Will require careful implementation of classification/rule population logic.
* **Performance:** Prediction logic needs to remain performant (threading).
* **Rule Structure Completeness:** Ensure `SourceRule` dataclasses hold all necessary *overridable* fields.
* **Preset Loading:** Robust preset loading/interpretation needed in `PredictionHandler`.
* **Static Config Loading:** Ensure the new `ProcessingEngine` correctly loads and uses the static `Configuration` object.
## 6. Documentation
This document (`ProjectNotes/Data_Flow_Refinement_Plan.md`) serves as the architectural plan. Relevant sections of the Developer Guide will need updating upon implementation.

65
ProjectNotes/GUI_Overhaul_Plan_Unified_View.md Normal file
View File

@@ -0,0 +1,65 @@
# GUI Overhaul Plan: Unified Hierarchical View
**Task:** Implement a UI overhaul for the Asset Processor Tool GUI to address usability issues and streamline the workflow for viewing and editing processing rules.
**Context:**
* A hierarchical rule system (`SourceRule`, `AssetRule`, `FileRule` in `rule_structure.py`) is used by the core engine (`asset_processor.py`).
* The current GUI (`gui/main_window.py`, `gui/rule_hierarchy_model.py`, `gui/rule_editor_widget.py`) uses a `QTreeView` for hierarchy, a separate `RuleEditorWidget` for editing selected items, and a `QTableView` (`PreviewTableModel`) for previewing file classifications.
* Relevant files analyzed: `gui/main_window.py`, `gui/rule_editor_widget.py`, `gui/rule_hierarchy_model.py`.
**Identified Issues with Current UI:**
1. **Window Resizing:** Selecting Source/Asset items causes window expansion because `RuleEditorWidget` displays large child lists (`assets`, `files`) as simple labels.
2. **GUI Not Updating on Add:** Potential regression where adding new inputs doesn't reliably update the preview/hierarchy.
3. **Incorrect Source Display:** Tree view shows "Source: None" instead of the input path (likely `SourceRule.input_path` is None when model receives it).
4. **Preview Table Stale:** Changes made in `RuleEditorWidget` (e.g., overrides) are not reflected in the `PreviewTableModel` because the `_on_rule_updated` slot in `main_window.py` doesn't trigger a refresh.
**Agreed-Upon Overhaul Plan:**
The goal is to create a more unified and streamlined experience by merging the hierarchy, editing overrides, and preview aspects into a single view, reducing redundancy.
1. **UI Structure Redesign:**
* **Left Panel:** Retain the existing Preset Editor panel (`main_window.py`'s `editor_panel`) for managing preset files (`.json`) and their complex rules (naming patterns, map type mappings, archetype rules, etc.).
* **Right Panel:** Replace the current three-part splitter (Hierarchy Tree, Rule Editor, Preview Table) with a **single Unified Hierarchical View**.
* Implementation: Use a `QTreeView` with a custom `QAbstractItemModel` and custom `QStyledItemDelegate`s for inline editing.
* Hierarchy Display: Show Input Source(s) -> Assets -> Files.
* Visual Cues: Use distinct background colors for rows representing Inputs, Assets, and Files.
2. **Unified View Columns & Functionality:**
* **Column 1: Name/Hierarchy:** Displays input path, asset name, or file name with indentation.
* **Column 2+: Editable Attributes (Context-Dependent):** Implement inline editors using delegates:
* **Input Row:** Optional editable field for `Supplier` override.
* **Asset Row:** `QComboBox` delegate for `Asset-Type` override (e.g., `GENERIC`, `DECAL`, `MODEL`).
* **File Row:**
* `QLineEdit` delegate for `Target Asset Name` override.
* `QComboBox` delegate for `Item-Type` override (e.g., `MAP-COL`, `MAP-NRM`, `EXTRA`, `MODEL_FILE`).
* **Column X: Status (Optional, Post-Processing):** Non-editable column showing processing status icon/text (Pending, Success, Warning, Error).
* **Column Y: Output Path (Optional, Post-Processing):** Non-editable column showing the final output path after successful processing.
3. **Data Flow and Initialization:**
* When inputs are added and a preset selected, `PredictionHandler` runs.
* `PredictionHandler` generates the `SourceRule` hierarchy *and* predicts initial `Asset-Type`, `Item-Type`, and `Target Asset Name`.
* The Unified View's model is populated with this `SourceRule`.
* *Initial values* in inline editors are set based on these *predicted* values.
* User edits in the Unified View directly modify attributes on the `SourceRule`, `AssetRule`, or `FileRule` objects held by the model.
4. **Dropdown Options Source:**
* Available options in dropdowns (`Asset-Type`, `Item-Type`) should be sourced from globally defined lists or Enums (e.g., in `rule_structure.py` or `config.py`).
5. **Addressing Original Issues (How the Plan Fixes Them):**
* **Window Resizing:** Resolved by removing `RuleEditorWidget`.
* **GUI Not Updating on Add:** Fix requires ensuring `add_input_paths` triggers `PredictionHandler` and updates the new Unified View model correctly.
* **Incorrect Source Display:** Fix requires ensuring `PredictionHandler` correctly populates `SourceRule.input_path`.
* **Preview Table Stale:** Resolved by merging preview/editing; edits are live in the main view.
**Implementation Tasks:**
* Modify `gui/main_window.py`: Remove the right-side splitter, `RuleEditorWidget`, `PreviewTableModel`/`View`. Instantiate the new Unified View. Adapt `add_input_paths`, `start_processing`, `_on_rule_hierarchy_ready`, etc., to interact with the new view/model.
* Create/Modify Model (`gui/rule_hierarchy_model.py` or new file): Implement a `QAbstractItemModel` supporting multiple columns, hierarchical data, and providing data/flags for inline editing.
* Create Delegates (`gui/delegates.py`?): Implement `QStyledItemDelegate` subclasses for `QComboBox` and `QLineEdit` editors in the tree view.
* Modify `gui/prediction_handler.py`: Ensure it predicts initial override values (`Asset-Type`, `Item-Type`, `Target Asset Name`) and includes them in the data passed back to the main window (likely within the `SourceRule` structure or alongside it). Ensure `SourceRule.input_path` is correctly set.
* Modify `gui/processing_handler.py`: Update it to potentially signal back status/output path updates that can be reflected in the new Unified View model's optional columns.
* Define Dropdown Sources: Add necessary Enums or lists to `rule_structure.py` or `config.py`.
This plan provides a clear path forward for implementing the UI overhaul.