Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Regression fixes and Documentation updates

Browse Source
This commit is contained in:
Rusfort 2025-05-01 10:45:32 +02:00
parent 6971b8189f
commit 26e1a769ce
10 changed files with 229 additions and 201 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
Documentation/02_Developer_Guide/01_Architecture.md
View File

@ -6,13 +6,13 @@ This document provides a high-level overview of the Asset Processor Tool's archi
The Asset Processor Tool is designed to process 3D asset source files into a standardized library format. Its high-level architecture consists of:
1. **Core Processing Engine (`AssetProcessor`):** The central component responsible for orchestrating the asset processing pipeline for a single input asset.
1. **Core Processing Engine (`processing_engine.py`):** The primary component responsible for executing the asset processing pipeline for a single input asset based on a provided `SourceRule` object and static configuration. The older `asset_processor.py` remains in the codebase for reference but is no longer used in the main processing flow.
2. **Configuration System (`Configuration`):** Handles loading core settings and merging them with supplier-specific rules defined in JSON presets.
3. **Multiple Interfaces:** Provides different ways to interact with the tool:
* Graphical User Interface (GUI)
* Command-Line Interface (CLI)
* Directory Monitor for automated processing.
These interfaces exchange data structures containing information about each file and asset set being processed.
The GUI now acts as the primary source of truth for processing rules, generating and managing the `SourceRule` hierarchy before sending it to the processing engine. The CLI and Monitor interfaces can also generate `SourceRule` objects to bypass the GUI for automated workflows.
4. **Optional Integration:** Includes scripts and logic for integrating with external software, specifically Blender, to automate material and node group creation.
## Hierarchical Rule System
@ -23,22 +23,27 @@ A key addition to the architecture is the **Hierarchical Rule System**, which pr
* **AssetRule:** Represents rules applied to a specific asset within a source (a source can contain multiple assets).
* **FileRule:** Represents rules applied to individual files within an asset.
This hierarchy allows for fine-grained control over processing parameters. When the processing engine needs a configuration value (e.g., a naming convention or image processing setting), it first checks the `FileRule` for the specific file, then the `AssetRule` for the asset containing the file, then the `SourceRule` for the overall source. If a value is not found at any of these dynamic levels, it falls back to the static configuration loaded from the selected preset. This prioritization logic (File > Asset > Source > Static Preset) ensures that the most specific rule available is always used.
This hierarchy allows for fine-grained control over processing parameters. The GUI's prediction logic generates this hierarchy with initial predicted values for overridable fields based on presets and file analysis. The processing engine then operates *solely* on the explicit values provided in this `SourceRule` object and static configuration, without internal prediction or fallback logic.
## Core Components
* `config.py`: Defines core, global settings and constants.
* `Presets/*.json`: Supplier-specific JSON files defining rules for file interpretation and processing.
* `configuration.py` (`Configuration` class): Loads `config.py` settings and merges them with a selected preset, pre-compiling regex patterns for efficiency.
* `asset_processor.py` (`AssetProcessor` class): Contains the core logic for processing a *single* asset through the defined pipeline steps. This component works with data structures containing detailed information about the asset and its files.
* `main.py`: The entry point for the Command-Line Interface (CLI). It handles argument parsing, logging, parallel processing orchestration, and triggering Blender scripts. It orchestrates the processing of multiple assets by interacting with the `AssetProcessor` for individual assets and manages the overall CLI execution flow.
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The GUI interacts with the core processing logic indirectly via dedicated handler classes (`ProcessingHandler`, `PredictionHandler`) running in separate threads. Data structures, such as input paths, configuration details, processing progress updates, and file prediction results, are passed between the GUI, these handlers, and the `AssetProcessor` using thread-safe mechanisms like Qt signals and slots.
* `config.py`: Defines core, global settings, constants, and centralized lists of allowed asset and file types.
* `Presets/*.json`: Supplier-specific JSON files defining rules for file interpretation and initial prediction.
* `configuration.py` (`Configuration` class): Loads `config.py` settings and merges them with a selected preset, pre-compiling regex patterns for efficiency. This static configuration is used by the processing engine.
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses used to represent the hierarchical processing rules.
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The GUI is responsible for generating and managing the `SourceRule` hierarchy via the Unified View and interacting with background handlers (`ProcessingHandler`, `PredictionHandler`).
* `unified_view_model.py`: Implements the `QAbstractItemModel` for the Unified Hierarchical View, holding the `SourceRule` data and handling inline editing.
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations for inline editing in the Unified View.
* `prediction_handler.py`: Generates the initial `SourceRule` hierarchy with predicted values based on input files and the selected preset.
* `processing_engine.py` (`ProcessingEngine` class): The new core component that executes the processing pipeline for a single `SourceRule` object using the static `Configuration`. It contains no internal prediction or fallback logic.
* `asset_processor.py` (`AssetProcessor` class): The older processing engine, kept for reference but not used in the main processing flow.
* `main.py`: The entry point for the Command-Line Interface (CLI). It handles argument parsing, logging, parallel processing orchestration, and triggering Blender scripts. It now orchestrates processing by passing `SourceRule` objects to the `ProcessingEngine`.
* `monitor.py`: Implements the directory monitoring feature using `watchdog`.
* `blenderscripts/`: Contains Python scripts designed to be executed *within* Blender for post-processing tasks.
## Processing Pipeline (Simplified)
The core processing engine (`AssetProcessor`) executes a series of steps for each asset:
The primary processing engine (`processing_engine.py`) executes a series of steps for each asset based on the provided `SourceRule` object and static configuration:
1. Extraction of input to a temporary workspace.
2. Classification of files (map, model, extra, ignored, unrecognised) using preset rules.
@ -51,6 +56,4 @@ The core processing engine (`AssetProcessor`) executes a series of steps for eac
9. Cleanup of the temporary workspace.
10. (Optional) Execution of Blender scripts for post-processing.
This architecture allows for a modular design, separating configuration, core processing logic, and different interfaces. The data structures flowing through this pipeline carry detailed information about individual files and asset sets. Examples include lists of input file paths, configuration objects, dictionaries summarizing processing outcomes, and detailed lists of file predictions. Parallel processing is utilized for efficiency, and background threads keep the GUI responsive.
**Note on Data Passing:** Major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. These changes are expected to involve new data structures and updated interaction patterns to convey detailed specifications for datasets/asset-sets and processing instructions for individual files. The documentation in this section, particularly regarding data flow, will require significant review and updates once the plan for these changes is finalized.
This architecture allows for a modular design, separating configuration, rule generation/management (primarily in the GUI), and core processing execution. The `SourceRule` object serves as a clear data contract between the GUI/prediction layer and the processing engine. Parallel processing is utilized for efficiency, and background threads keep the GUI responsive.

40
Documentation/02_Developer_Guide/02_Codebase_Structure.md
View File

@ -4,19 +4,21 @@ This document outlines the key files and directories within the Asset Processor
```
Asset_processor_tool/
├── asset_processor.py # Core class handling single asset processing pipeline
├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)
├── asset_processor.py # Older core class, kept for reference (not used in main flow)
├── config.py # Core settings, constants, and allowed types
├── configuration.py # Class for loading and accessing configuration (merges config.py and presets)
├── detailed_documentation_plan.md # (Existing file, potentially outdated)
├── Dockerfile # Instructions for building the Docker container image
├── documentation_plan.md # Plan for the new documentation structure (this plan)
├── documentation.txt # Original developer documentation (to be migrated)
├── main.py # CLI Entry Point & processing orchestrator
├── main.py # CLI Entry Point & processing orchestrator (calls processing_engine)
├── monitor.py # Directory monitoring script for automated processing
├── processing_engine.py # New core class handling single asset processing based on SourceRule
├── readme.md # Original main documentation file (to be migrated)
├── readme.md.bak # Backup of readme.md
├── requirements-docker.txt # Dependencies specifically for the Docker environment
├── requirements.txt # Python package dependencies for standard execution
├── rule_structure.py # Dataclasses for hierarchical rules (SourceRule, AssetRule, FileRule)
├── blenderscripts/ # Scripts for integration with Blender
│ ├── create_materials.py # Script to create materials linking to node groups
│ └── create_nodegroups.py # Script to create node groups from processed assets
@ -29,10 +31,11 @@ Asset_processor_tool/
│ ├── 01_User_Guide/
│ └── 02_Developer_Guide/
├── gui/ # Contains files related to the Graphical User Interface
│ ├── delegates.py # Custom delegates for inline editing in Unified View
│ ├── main_window.py # Main GUI application window and layout
│ ├── processing_handler.py # Handles background processing logic for the GUI
│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI
│ ├── preview_table_model.py # Model and proxy for the GUI's preview table
│ ├── prediction_handler.py # Generates initial SourceRule hierarchy with predictions
│ ├── unified_view_model.py # Model for the Unified Hierarchical View
│ └── ... # Other GUI components
├── Presets/ # Preset definition files
│ ├── _template.json # Template for creating new presets
@ -46,19 +49,20 @@ Asset_processor_tool/
**Key Files and Directories:**
* `asset_processor.py`: Contains the `AssetProcessor` class, the core logic for processing a single asset through the pipeline. Includes methods for classification, map processing, merging, metadata generation, and output organization. Also provides methods for predicting output structure used by the GUI.
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config.py` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency.
* `config.py`: Stores global default settings, constants, and core rules (e.g., standard map types, default resolutions, merge rules, output format rules, Blender paths).
* `main.py`: Entry point for the Command-Line Interface (CLI). Handles argument parsing, logging setup, parallel processing orchestration (using `concurrent.futures.ProcessPoolExecutor`), calls `AssetProcessor` via a wrapper function, and optionally triggers Blender scripts.
* `asset_processor.py`: Contains the older `AssetProcessor` class. It is kept for reference but is no longer used in the main processing flow orchestrated by `main.py` or the GUI.
* `config.py`: Stores global default settings, constants, core rules, and centralized lists of `ALLOWED_ASSET_TYPES` and `ALLOWED_FILE_TYPES` used for validation and GUI dropdowns.
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config.py` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency. An instance of this class is passed to the `ProcessingEngine`.
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses. These structures represent the hierarchical processing rules and are the primary data contract passed from the GUI/prediction layer to the processing engine.
* `processing_engine.py`: Defines the new `ProcessingEngine` class. This is the core component that executes the processing pipeline for a single asset based *solely* on a provided `SourceRule` object and the static `Configuration`. It contains no internal prediction or fallback logic.
* `main.py`: Entry point for the Command-Line Interface (CLI). It handles argument parsing, logging setup, parallel processing orchestration (using `concurrent.futures.ProcessPoolExecutor`), and triggering Blender scripts. It now orchestrates processing by generating or receiving `SourceRule` objects and passing them to the `ProcessingEngine`.
* `monitor.py`: Implements the automated directory monitoring feature using the `watchdog` library. Contains the `ZipHandler` class to detect new ZIP files and trigger processing via `main.run_processing`.
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6.
* `main_window.py`: Defines the `MainWindow` class, the main application window structure, UI layout, event handling, and menu setup. Manages GUI-specific logging (`QtLogHandler`).
* `processing_handler.py`: Defines the `ProcessingHandler` class (runs on a `QThread`). Manages the execution of the main asset processing pipeline and Blender script execution in the background.
* `prediction_handler.py`: Defines the `PredictionHandler` class (runs on a `QThread`). Manages background file analysis/preview generation.
* `preview_table_model.py`: Defines `PreviewTableModel` and `PreviewSortFilterProxyModel` for managing and displaying data in the GUI's preview table.
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6. The GUI is responsible for managing user input, generating and editing the `SourceRule` hierarchy, and interacting with background handlers.
* `main_window.py`: Defines the `MainWindow` class, the main application window structure, UI layout, event handling, and menu setup. Integrates the Unified Hierarchical View. Manages GUI-specific logging (`QtLogHandler`).
* `unified_view_model.py`: Implements the `QAbstractItemModel` for the Unified Hierarchical View (`QTreeView`). It holds the `SourceRule` hierarchy and provides data and flags for display and inline editing.
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations (e.g., for `QComboBox`, `QLineEdit`) used by the Unified View to provide inline editors for rule attributes.
* `processing_handler.py`: Defines the `ProcessingHandler` class (runs on a `QThread`). Manages the execution of the `ProcessingEngine` in background processes and communicates status/results back to the GUI.
* `prediction_handler.py`: Defines the `PredictionHandler` class (runs on a `QThread`). Generates the initial `SourceRule` hierarchy with predicted values based on input files and the selected preset. Emits a signal with the generated `SourceRule` list for the GUI.
* `blenderscripts/`: Contains Python scripts (`create_nodegroups.py`, `create_materials.py`) designed to be executed *within* Blender for post-processing.
* `Presets/`: Contains supplier-specific configuration files in JSON format.
* `Presets/`: Contains supplier-specific configuration files in JSON format, used by the `PredictionHandler` for initial rule generation.
* `Testfiles/`: Contains example input assets for testing purposes.
* `Tickets/`: Directory for issue and feature tracking using Markdown files.
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `asset_processor` modules are currently being planned. The descriptions of module interactions and data flow within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
* `Tickets/`: Directory for issue and feature tracking using Markdown files.

94
Documentation/02_Developer_Guide/03_Key_Components.md
View File

@ -2,22 +2,21 @@
This document describes the major classes and modules that form the core of the Asset Processor Tool.
## `AssetProcessor` (`asset_processor.py`)
## `ProcessingEngine` (`processing_engine.py`)
The `AssetProcessor` class is the central engine of the tool. It is responsible for processing a *single* input asset (either a ZIP archive or a folder) through the entire pipeline. Its key responsibilities include:
The `ProcessingEngine` class is the new core component responsible for executing the asset processing pipeline for a *single* input asset. Unlike the older `AssetProcessor`, this engine operates *solely* based on a complete `SourceRule` object provided to its `process()` method and the static `Configuration` object passed during initialization. It contains no internal prediction, classification, or fallback logic. Its key responsibilities include:
* Setting up and cleaning up a temporary workspace for processing.
* Extracting or copying input files to the workspace.
* Inventorying and classifying files based on configured rules (maps, models, extra, ignored, unrecognised).
* Determining asset metadata such as name, category, and archetype.
* Processing texture maps (resizing, format/bit depth conversion, handling Gloss->Roughness inversion, calculating statistics).
* Merging channels from different maps according to merge rules.
* Generating the `metadata.json` file containing details about the processed asset.
* Processing files based on the explicit rules and predicted values contained within the input `SourceRule`.
* Processing texture maps (resizing, format/bit depth conversion, inversion, stats calculation) using parameters from the `SourceRule` or static `Configuration`.
* Merging channels based on rules defined in the static `Configuration` and parameters from the `SourceRule`.
* Generating the `metadata.json` file containing details about the processed asset, incorporating information from the `SourceRule`.
* Organizing the final output files into the structured library directory.
* Accepting a `SourceRule` object which represents the hierarchical rules (Source, Asset, File) that can override static configuration values.
* Implementing the logic (`_get_rule_with_fallback`) to retrieve configuration values by prioritizing rules in the order: File -> Asset -> Source -> Static Config.
* Applying this hierarchical rule logic during file classification, prediction, image processing, and metadata generation.
* Providing methods (`get_detailed_file_predictions`) used by the GUI for previewing file classification, now incorporating the hierarchical rules.
## `AssetProcessor` (`asset_processor.py`)
The `AssetProcessor` class is the older processing engine. It is kept in the codebase for reference but is **no longer used** in the main processing flow orchestrated by `main.py` or the GUI. Its original role was similar to the new `ProcessingEngine`, but it included internal prediction, classification, and fallback logic based on hierarchical rules and static configuration.
## `Rule Structure` (`rule_structure.py`)
@ -27,19 +26,7 @@ This module defines the data structures used to represent the hierarchical proce
* `AssetRule`: A dataclass representing rules applied at the asset level. It contains nested `FileRule` objects.
* `FileRule`: A dataclass representing rules applied at the file level.
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `map_type_override`) and support serialization (Pickle, JSON) to allow them to be passed between different parts of the application, including across process boundaries.
## `Rule Hierarchy Model` (`gui/rule_hierarchy_model.py`)
The `RuleHierarchyModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
* Wrap a `SourceRule` object and expose its hierarchical structure (Source -> Asset -> File) to a `QTreeView`.
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy.
* Enable navigation and selection within the rule hierarchy in the GUI.
## `Rule Editor Widget` (`gui/rule_editor_widget.py`)
The `RuleEditorWidget` is a custom Qt widget that provides a user interface for viewing and editing the attributes of a selected rule object (`SourceRule`, `AssetRule`, or `FileRule`). It dynamically generates input fields (e.g., line edits, checkboxes) based on the attributes of the rule object it is currently displaying. It allows users to modify the dynamic rule parameters through the GUI.
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `map_type_override`). Attributes like `asset_type` and `item_type_override` now use string types, which are validated against centralized lists in `config.py`. These structures support serialization (Pickle, JSON) to allow them to be passed between different parts of the application, including across process boundaries.
## `Configuration` (`configuration.py`)
@ -49,37 +36,58 @@ The `Configuration` class manages the tool's settings. It is responsible for:
* Loading the supplier-specific rules from a selected preset JSON file (`Presets/*.json`).
* Merging the core settings and preset rules into a single, unified configuration object.
* Validating the loaded configuration to ensure required settings are present.
* Pre-compiling regular expression patterns defined in the preset for efficient file classification by the `AssetProcessor`.
* Pre-compiling regular expression patterns defined in the preset for efficient file classification by the `PredictionHandler`.
An instance of the `Configuration` class is typically created once per asset processing task (within a worker process) to ensure isolated and correct settings for each asset.
An instance of the `Configuration` class is typically created once per application run (or per processing batch) and passed to the `ProcessingEngine`.
## `MainWindow` (`gui/main_window.py`)
The `MainWindow` class is the main application window for the Graphical User Interface (GUI). It handles the overall UI layout and user interaction:
* Sets up the main window structure, including panels for the preset editor and processing controls.
* Manages the layout of UI elements like the drag-and-drop area, preview table, buttons, and status bar.
* Connects user actions (button clicks, drag/drop events) to corresponding handler methods (slots).
* Interacts with background processing and prediction handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to update the UI safely from background threads.
* Manages the GUI-specific logging handler (`QtLogHandler`) to display logs in the UI console.
* Defines the main application window structure and layout using PySide6 widgets.
* Arranges the Preset Editor panel (left) and the Unified Hierarchical View (right).
* Setting up the menu bar, including the "View" menu for toggling the Log Console.
* Connecting user interactions (button clicks, drag-and-drop events, edits in the Unified View) to corresponding methods (slots) within the `MainWindow` or other handler classes.
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
* Receiving the initial `SourceRule` hierarchy from the `PredictionHandler` and populating the `UnifiedViewModel`.
* Sending the final, potentially user-modified, `SourceRule` list to `main.py` to initiate processing via the `ProcessingEngine`.
## `Unified View Model` (`gui/unified_view_model.py`)
The `UnifiedViewModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
* Wrap a list of `SourceRule` objects and expose their hierarchical structure (Source -> Asset -> File) to a `QTreeView` (the Unified Hierarchical View).
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`, `flags`, `setData`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy and support inline editing of specific attributes (e.g., asset type, item type override, target asset name override).
* Hold the `SourceRule` data that is the single source of truth for the GUI's processing rules.
## `Delegates` (`gui/delegates.py`)
This module contains custom `QStyledItemDelegate` implementations used by the Unified Hierarchical View (`QTreeView`) to provide inline editors for specific data types or rule attributes. Examples include delegates for:
* `QComboBox`: For selecting from a predefined list of options (e.g., allowed asset types, allowed file types sourced from `config.py`).
* `QLineEdit`: For free-form text editing (e.g., target asset name override, supplier identifier override).
These delegates handle the presentation and editing of data within the tree view cells, interacting with the `UnifiedViewModel` to get and set data.
## `ProcessingHandler` (`gui/processing_handler.py`)
The `ProcessingHandler` class is designed to run in a separate `QThread` within the GUI. Its purpose is to manage the execution of the main asset processing pipeline and optional Blender scripts in the background, preventing the GUI from freezing. It:
The `ProcessingHandler` class is designed to run in a separate `QThread` within the GUI. Its purpose is to manage the execution of the main asset processing pipeline using the **`ProcessingEngine`** in the background, preventing the GUI from freezing. It:
* Manages a `concurrent.futures.ProcessPoolExecutor` to run individual asset processing tasks (`AssetProcessor.process()`) in separate worker processes.
* Submits processing tasks to the pool and monitors their completion.
* Communicates progress, status updates, and results back to the `MainWindow` using Qt signals.
* Handles the execution of Blender scripts via subprocess calls after asset processing is complete.
* Provides logic for cancelling ongoing processing tasks (though cancellation of already running worker processes is not immediate).
* Manages a `concurrent.futures.ProcessPoolExecutor` to run individual asset processing tasks (`ProcessingEngine.process()`) in separate worker processes.
* Submits processing tasks to the pool, passing the relevant `SourceRule` object and `Configuration` instance to the `ProcessingEngine`.
* Monitors task completion and communicates progress, status updates, and results back to the `MainWindow` using Qt signals.
* Handles the execution of optional Blender scripts via subprocess calls after asset processing is complete.
* Provides logic for cancelling ongoing processing tasks.
## `PredictionHandler` (`gui/prediction_handler.py`)
The `PredictionHandler` class also runs in a separate `QThread` in the GUI. It is responsible for generating file classification previews in the background without blocking the UI. It:
The `PredictionHandler` class also runs in a separate `QThread` in the GUI. It is responsible for generating the initial `SourceRule` hierarchy with predicted values based on the input files and the selected preset. It:
* Calls methods on the `AssetProcessor` (specifically `get_detailed_file_predictions`) to analyze input files and predict their classification and output names based on the selected processing preset.
* Uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks.
* Sends the prediction results back to the `MainWindow` via Qt signals to update the preview table.
* Takes a list of input files and the selected preset name as input.
* Uses logic (including accessing preset rules and `config.py`'s allowed types) to analyze files and predict initial values for overridable fields in the `SourceRule`, `AssetRule`, and `FileRule` objects (e.g., asset type, item type, target asset name).
* Constructs the complete `SourceRule` hierarchy based on these predictions.
* Emits a signal (`rule_hierarchy_ready`) with the generated `List[SourceRule]` to the `MainWindow` to populate the Unified Hierarchical View.
## `ZipHandler` (`monitor.py`)
@ -90,6 +98,4 @@ The `ZipHandler` is a custom event handler used by the `monitor.py` script, buil
* Triggering the main asset processing logic (`main.run_processing`) for valid new ZIP files.
* Managing the movement of processed source ZIP files to 'processed' or 'error' directories.
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness.
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of module interactions and data flow within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness. The `SourceRule` object serves as a clear data contract between the GUI/prediction layer and the processing engine.

47
Documentation/02_Developer_Guide/05_Processing_Pipeline.md
View File

@ -1,8 +1,8 @@
# Developer Guide: Processing Pipeline
This document details the step-by-step technical process executed by the `AssetProcessor` class (`asset_processor.py`) when processing a single asset.
This document details the step-by-step technical process executed by the `ProcessingEngine` class (`processing_engine.py`) when processing a single asset.
The `AssetProcessor.process()` method orchestrates the following pipeline. A key aspect of this pipeline is the integration of the **Hierarchical Rule System**, which allows dynamic rules (Source, Asset, File) to override static configuration values loaded from presets. The `_get_rule_with_fallback` helper method is used throughout the pipeline to retrieve configuration values, prioritizing rules in the order: File -> Asset -> Source -> Static Config.
The `ProcessingEngine.process()` method orchestrates the following pipeline based *solely* on the provided `SourceRule` object and the static `Configuration` object passed during engine initialization. It contains no internal prediction, classification, or fallback logic. All necessary overrides and static configuration values are accessed directly from these inputs.
The pipeline steps are:
@ -16,61 +16,44 @@ The pipeline steps are:
3. **File Inventory and Classification (`_inventory_and_classify_files`)**:
* Scans the contents of the temporary workspace.
* Uses the pre-compiled regex patterns from the loaded `Configuration` object and the rules from the `SourceRule` object to classify each file. The hierarchical rules can influence classification patterns and other parameters used in this step.
* Classification follows a multi-pass approach for priority:
* Explicitly marked `Extra/` files (using `move_to_extra_patterns` regex).
* Model files (using `model_patterns` regex).
* Potential Texture Maps (matching `map_type_mapping` keyword patterns).
* Standalone 16-bit variants check (using `bit_depth_variants` patterns).
* Prioritization of 16-bit variants over their 8-bit counterparts (marking the 8-bit version as `Ignored`).
* Final classification of remaining potential maps.
* Remaining files are classified as `Unrecognised` (and typically moved to `Extra/` later).
* Uses the pre-compiled regex patterns from the loaded `Configuration` object and the explicit rules and predicted classifications from the input `SourceRule` object to classify each file. The classification is based on the data already determined by the `PredictionHandler` and potentially modified by the user in the GUI.
* Stores the classification results (including source path, determined map type, potential variant suffix, etc.) in `self.classified_files`.
* Sorts potential map variants based on preset rule order, keyword order within the rule, and finally alphabetical path to determine suffix assignment priority (`-1`, `-2`, etc.).
* Sorts potential map variants based on the order provided in the `SourceRule` or static configuration.
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
* This step now utilizes the hierarchical rules from the `SourceRule` object. For example, the `supplier_identifier`, `asset_type`, and `asset_name_override` attributes from the rules can override the values determined from the static configuration or input file names. The `_get_rule_with_fallback` method is used here to apply the rule prioritization logic.
* Determines the base asset name using `source_naming_convention` rules from the `Configuration` (separators, indices), with fallbacks to common prefixes or the input name. Handles multiple distinct assets within a single input source.
* Determines the asset category (`Texture`, `Asset`, `Decal`) based on the presence of model files or `decal_keywords` in the `Configuration`.
* Determines the asset archetype (e.g., `Wood`, `Metal`) by matching keywords from `archetype_rules` (in `Configuration`) against file stems or the determined base name.
* Stores this preliminary metadata.
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` object and the static configuration from the `Configuration` object. Overrides (like `supplier_identifier`, `asset_type`, and `asset_name_override`) are taken directly from the `SourceRule`.
5. **Skip Check**:
* If the `overwrite` flag (passed during initialization) is `False`, the tool checks if the final output directory for the determined asset name already exists and contains a `metadata.json` file.
* If both exist, processing for this specific asset is skipped, marked as "skipped", and the pipeline moves to the next asset (if processing multiple assets from one source) or finishes.
6. **Map Processing (`_process_maps`)**:
* Iterates through the files classified as texture maps for the current asset. Configuration values used in this step, such as target resolutions, bit depth rules, and output format rules, are now retrieved using the `_get_rule_with_fallback` method, allowing them to be overridden by the hierarchical rules.
* Iterates through the files classified as texture maps for the current asset based on the `SourceRule`. Configuration values used in this step, such as target resolutions, bit depth rules, and output format rules, are retrieved directly from the static `Configuration` object or explicit overrides in the `SourceRule`.
* Loads the image using `cv2.imread` (handling grayscale and unchanged flags). Converts BGR to RGB internally for consistency (except for saving non-EXR formats).
* Handles Glossiness-to-Roughness inversion if necessary (loads gloss, inverts `1.0 - img/norm`, prioritizes gloss source if both exist).
* Resizes the image to target resolutions defined in `IMAGE_RESOULTIONS` (from `Configuration`) using `cv2.resize` (`INTER_LANCZOS4` for downscaling). Upscaling is generally avoided by checks.
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` vs `force_8bit`).
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a hierarchy of rules:
* `FORCE_LOSSLESS_MAP_TYPES` list (overrides other logic).
* `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for large 8-bit maps).
* Source format, target bit depth, and configured defaults (`OUTPUT_FORMAT_16BIT_PRIMARY`, `OUTPUT_FORMAT_8BIT`).
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (from `Configuration`) or overrides in the `SourceRule`.
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a hierarchy of rules defined in the `Configuration` or overrides in the `SourceRule`.
* Converts the NumPy array data type appropriately before saving (e.g., float to uint8/uint16 with scaling).
* Saves the processed map using `cv2.imwrite` (converting RGB back to BGR if saving to non-EXR formats). Includes fallback logic (e.g., attempting PNG if saving 16-bit EXR fails).
* Calculates image statistics (Min/Max/Mean) using `_calculate_image_stats` on normalized float64 data for the `CALCULATE_STATS_RESOLUTION`.
* Calculates image statistics (Min/Max/Mean) using `_calculate_image_stats` on normalized float64 data for the `CALCULATE_STATS_RESOLUTION` (from `Configuration`).
* Determines the aspect ratio change string (e.g., `"EVEN"`, `"X150"`) using `_normalize_aspect_ratio_change`.
* Stores details about each processed map (path, resolution, format, stats, etc.) in `processed_maps_details_asset`.
7. **Map Merging (`_merge_maps_from_source`)**:
* The `MAP_MERGE_RULES` themselves can be influenced by the hierarchical rules. Additionally, parameters within the merge rules, such as default channel values or output format/bit depth, are retrieved using `_get_rule_with_fallback`, allowing dynamic overrides.
* Iterates through the `MAP_MERGE_RULES` defined in the `Configuration`.
* Identifies the required *source* map files needed as input for each merge rule based on the classified files.
* Identifies the required *source* map files needed as input for each merge rule based on the classified files in the `SourceRule`.
* Determines common resolutions available across the required input maps.
* Loads the necessary source map channels for each common resolution (using a helper `_load_and_transform_source` which includes caching).
* Converts inputs to normalized float32 (0-1).
* Injects default channel values (from rule `defaults`) if an input channel is missing.
* Injects default channel values (from rule `defaults` in `Configuration` or overrides in `SourceRule`) if an input channel is missing.
* Merges channels using `cv2.merge`.
* Determines output bit depth and format based on rules (similar logic to `_process_maps`, considering input properties). Handles potential JPG 16-bit conflict by forcing 8-bit.
* Determines output bit depth and format based on rules in `Configuration` or overrides in `SourceRule`. Handles potential JPG 16-bit conflict by forcing 8-bit.
* Saves the merged map using the `_save_image` helper (includes data type/color space conversions and fallback).
* Stores details about each merged map in `merged_maps_details_asset`.
8. **Metadata File Generation (`_generate_metadata_file`)**:
* The metadata generated now includes information derived from the applied hierarchical rules, in addition to the static configuration and processing results.
* Collects all determined information for the current asset: base metadata, details from `processed_maps_details_asset` and `merged_maps_details_asset`, list of ignored files, source preset used, etc.
* Collects all determined information for the current asset: base metadata, details from `processed_maps_details_asset` and `merged_maps_details_asset`, list of ignored files, source preset used, etc. This information is derived from the input `SourceRule` and the processing results.
* Writes this collected data into the `metadata.json` file within the temporary workspace using `json.dump`.
9. **Output Organization (`_organize_output_files`)**:
@ -82,6 +65,6 @@ The pipeline steps are:
* Removes the temporary workspace directory and its contents using `shutil.rmtree()`. This is called within a `finally` block to ensure cleanup is attempted even if errors occur during processing.
11. **(Optional) Blender Script Execution**:
* If triggered via CLI arguments (`--nodegroup-blend`, `--materials-blend`) or GUI controls, the orchestrator (`main.py` or `gui/processing_handler.py`) executes the corresponding Blender scripts (`blenderscripts/*.py`) using `subprocess.run` after the `AssetProcessor.process()` call completes successfully for an asset batch. See `Developer Guide: Blender Integration Internals` for more details.
* If triggered via CLI arguments (`--nodegroup-blend`, `--materials-blend`) or GUI controls, the orchestrator (`main.py` or `gui/processing_handler.py`) executes the corresponding Blender scripts (`blenderscripts/*.py`) using `subprocess.run` after the `ProcessingEngine.process()` call completes successfully for an asset batch. See `Developer Guide: Blender Integration Internals` for more details.
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how data is processed and transformed within this pipeline reflect the current state and will require review and updates once the plan for these changes is finalized.
This pipeline, executed by the `ProcessingEngine`, provides a clear and explicit processing flow based on the complete rule set provided by the GUI or other interfaces.

95
Documentation/02_Developer_Guide/06_GUI_Internals.md
View File

@ -11,20 +11,20 @@ The GUI is built using `PySide6`, which provides Python bindings for the Qt fram
The `MainWindow` class is the central component of the GUI application. It is responsible for:
* Defining the main application window structure and layout using PySide6 widgets.
* Arranging the Preset Editor panel (left) and the Processing panel (right).
* Setting up the menu bar, including the "View" menu for toggling the Log Console and Detailed File Preview.
* Connecting user interactions (button clicks, drag-and-drop events, checkbox states, spinbox values) to corresponding methods (slots) within the `MainWindow` or other handler classes.
* Arranging the Preset Editor panel (left) and the **Unified Hierarchical View** (right).
* Setting up the menu bar, including the "View" menu for toggling the Log Console.
* Connecting user interactions (button clicks, drag-and-drop events, edits in the Unified View) to corresponding methods (slots) within the `MainWindow` or other handler classes.
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
* Integrating the `QTreeView` (displaying the rule hierarchy via `RuleHierarchyModel`) and the `RuleEditorWidget` for interactive rule editing.
* Connecting the selection changes in the `QTreeView` to update the `RuleEditorWidget` with the selected rule object's attributes.
* Receiving the initial `SourceRule` hierarchy from the `PredictionHandler` and populating the `UnifiedViewModel`.
* Sending the final, potentially user-modified, `SourceRule` list to `main.py` to initiate processing via the `ProcessingEngine`.
## Threading and Background Tasks
To keep the UI responsive during intensive operations like asset processing and file preview generation, the GUI utilizes background threads managed by `QThread`.
To keep the UI responsive during intensive operations like asset processing and rule prediction, the GUI utilizes background threads managed by `QThread`.
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool and monitors their completion. It uses Qt signals to communicate progress updates, file status changes, and overall processing completion back to the `MainWindow` on the main UI thread. It also handles the execution of optional Blender scripts via subprocess calls after processing. This handler processes and utilizes data structures received from the core processing engine, such as status summaries. It receives the initial `SourceRule` object generated by the `PredictionHandler` and passes it to the `AssetProcessor` for processing.
* **`PredictionHandler` (`gui/prediction_handler.py`):** This class also runs in a separate `QThread`. It is responsible for generating the detailed file classification previews displayed in the preview table. It calls methods on the `AssetProcessor` (`get_detailed_file_predictions`) to perform the analysis in the background. It uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks. Results are sent back to the `MainWindow` via Qt signals to update the preview table data. This handler works with data structures containing file prediction details.
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline using the **`ProcessingEngine`** for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool, passing the relevant `SourceRule` object and `Configuration` instance to the `ProcessingEngine`. It monitors task completion and communicates progress, status updates, and results back to the `MainWindow` on the main UI thread using Qt signals. It also handles the execution of optional Blender scripts via subprocess calls after processing.
* **`PredictionHandler` (`gui/prediction_handler.py`):** This class also runs in a separate `QThread`. It is responsible for generating the initial `SourceRule` hierarchy with predicted values based on the input files and the selected preset. It uses logic (including accessing preset rules and `config.py`'s allowed types) to analyze files and predict initial values for overridable fields in the `SourceRule`, `AssetRule`, and `FileRule` objects (e.g., asset type, item type, target asset name). It constructs the complete `SourceRule` hierarchy based on these predictions and emits a signal (`rule_hierarchy_ready`) with the generated `List[SourceRule]` to the `MainWindow` to populate the Unified Hierarchical View.
## Communication (Signals and Slots)
@ -37,75 +37,34 @@ Communication between the main UI thread (`MainWindow`) and the background threa
The GUI includes an integrated preset editor panel. This allows users to interactively create, load, modify, and save preset `.json` files directly within the application. The editor typically uses standard UI widgets to display and edit the key fields of the preset structure.
## Preview Table
## Unified Hierarchical View (`gui/unified_view_model.py`, `gui/delegates.py`)
The core of the GUI's rule editing interface is the Unified Hierarchical View, implemented using a `QTreeView` with a custom model and delegates.
The `PreviewTableModel` receives a list of file prediction dictionaries from the `PredictionHandler` via the `prediction_results_ready` signal. This list contains dictionaries for each file with details such as original path, predicted asset name, status, and other relevant information.
* **`Unified View Model` (`gui/unified_view_model.py`):** This class implements a `QAbstractItemModel` to expose the structure of a list of `SourceRule` objects (Source -> Asset -> File) to the `QTreeView`. It holds the `SourceRule` data that is the single source of truth for the GUI's processing rules. It provides data and flags for display in multiple columns and supports inline editing of specific rule attributes (e.g., asset type, item type override, target asset name override) by interacting with delegates.
* **`Delegates` (`gui/delegates.py`):** This module contains custom `QStyledItemDelegate` implementations used by the `QTreeView` to provide inline editors for specific data types or rule attributes. Examples include delegates for `QComboBox` (for selecting from allowed types sourced from `config.py`) and `QLineEdit` (for free-form text editing). These delegates handle the presentation and editing of data within the tree view cells, interacting with the `UnifiedViewModel` to get and set data.
The `PreviewTableModel` is designed to process and display this file prediction data. Instead of directly displaying the flat list, it processes and transforms the data into a structured list of rows (`self._table_rows`). This transformation involves:
The `PredictionHandler` generates the initial `SourceRule` hierarchy, which is then set on the `UnifiedViewModel`. The `QTreeView` displays this model, allowing users to navigate the hierarchy and make inline edits to the rule attributes. Edits made in the view directly modify the attributes of the underlying rule objects in the `SourceRule` hierarchy held by the model.
1. **Grouping:** Files are grouped based on their `source_asset`.
2. **Separation:** Within each asset group, files are separated into `main_files` (Mapped, Model, Error) and `additional_files` (Ignored, Extra, Unrecognised, Unmatched Extra).
3. **Structuring Rows:** Rows are created for `self._table_rows` to represent the grouped data. Each row can contain information about a main file and/or an additional file, allowing for the display of additional files in a separate column aligned with the main files of the same asset. Empty rows are created if there are more additional files than main files for an asset to maintain alignment.
The `data()` method of the `PreviewTableModel` then accesses this structured `self._table_rows` list to provide data to the `QTableView` for display. It handles different columns and roles (Display, Tooltip, Foreground, and Background).
* `Qt.ItemDataRole.ForegroundRole`: Used to set the text color of individual cells based on the status of the file they represent. Coloring is applied to cells corresponding to a main file based on the main file's status, and to cells in the "Additional Files" column based on the additional file's status.
* `Qt.ItemDataRole.BackgroundRole`: Used to provide alternating background colors based on the index of the asset group the row belongs to in a sorted list of unique assets, improving visual separation between different asset groups.
The `PreviewSortFilterProxyModel` operates on this structured data, implementing a multi-level sort based on source asset, row type (main vs. additional-only), and file paths within those types.
* Calls methods on the `AssetProcessor` (specifically `get_detailed_file_predictions`) to perform the analysis in the background. It uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks. Results are sent back to the `MainWindow` via Qt signals to update the preview table data. This handler works with data structures containing file prediction details. It is also responsible for generating the initial `SourceRule` hierarchy based on the input files and the selected preset, and emitting a signal to provide this `SourceRule` object to the `MainWindow`.
## Rule Hierarchy UI (`gui/rule_hierarchy_model.py`, `gui/rule_editor_widget.py`)
The GUI now includes dedicated components for visualizing and editing the hierarchical processing rules:
* **`Rule Hierarchy Model` (`gui/rule_hierarchy_model.py`):** This class implements `QAbstractItemModel` to expose the structure of a `SourceRule` object (Source -> Asset -> File) to a `QTreeView`. It allows the `QTreeView` to display the hierarchy and enables user selection of individual rule objects (SourceRule, AssetRule, or FileRule).
* **`Rule Editor Widget` (`gui/rule_editor_widget.py`):** This custom widget provides a dynamic form for editing the attributes of the currently selected rule object from the hierarchy tree. When a rule object is selected in the `QTreeView`, the `MainWindow` updates the `RuleEditorWidget` to display and allow modification of that object's specific parameters.
These components are integrated into the `MainWindow`. The `PredictionHandler` generates the initial `SourceRule` hierarchy, which is then set on the `RuleHierarchyModel`. The `QTreeView` displays this model, and selection changes in the tree trigger updates to the `RuleEditorWidget`, allowing users to interactively modify the dynamic rules before processing. Edits made in the `RuleEditorWidget` directly modify the attributes of the underlying rule objects in the `SourceRule` hierarchy.
### Preview Table Column Configuration
The display and behavior of the columns in the `QTableView` are configured in `gui/main_window.py`. The current configuration is as follows:
* **Column Order (from left to right):**
1. Status
2. Predicted Asset
3. Details
4. Original Path
5. Additional Files
* **Column Resizing:**
* Status: Resizes to content.
* Predicted Asset: Resizes to content.
* Details: Resizes to content.
* Original Path: Resizes to content (fixed width behavior).
* Additional Files: Stretches to fill available space.
**Data Flow Diagram:**
**Data Flow Diagram (GUI Rule Management):**
```mermaid
graph LR
A[PredictionHandler] -- prediction_results_ready(flat_list) --> B(PreviewTableModel);
subgraph PreviewTableModel
C[set_data] -- Processes flat_list --> D{Internal Grouping & Transformation};
D -- Creates --> E[_table_rows (Structured List)];
F[data()] -- Reads from --> E;
end
B -- Provides data via data() --> G(QTableView via Proxy);
style B fill:#f9f,stroke:#333,stroke-width:2px
style C fill:#ccf,stroke:#333,stroke-width:1px
style D fill:#lightgrey,stroke:#333,stroke-width:1px
style E fill:#ccf,stroke:#333,stroke-width:1px
style F fill:#ccf,stroke:#333,stroke-width:1px
A[User Input (Drag/Drop, Preset Select)] --> B(MainWindow);
B -- Calls --> C(PredictionHandler);
C -- Generates SourceRule Hierarchy with Predictions --> D(UnifiedViewModel);
B -- Sets Model --> E(QTreeView - Unified View);
E -- Displays Data from --> D;
E -- Uses Delegates from --> F(Delegates);
F -- Interact with --> D;
User -- Edits Rules via --> E;
E -- Updates Data in --> D;
B -- Triggers Processing with Final SourceRule List --> G(main.py / ProcessingHandler);
```
### Application Styling
## Application Styling
The application style is explicitly set to 'Fusion' in `gui/main_window.py` to provide a more consistent look and feel across different operating systems, particularly to address styling inconsistencies observed on Windows 11. A custom `QPalette` is also applied to the application to adjust default colors within the 'Fusion' style, specifically to change the background color of list-like widgets and potentially other elements from a default dark blue to a more neutral grey.
The application style is explicitly set to 'Fusion' in `gui/main_window.py` to provide a more consistent look and feel across different operating systems. A custom `QPalette` is also applied to the application to adjust default colors within the 'Fusion' style.
## Logging
@ -115,4 +74,4 @@ A custom `QtLogHandler` is used to redirect log messages from the standard Pytho
The GUI provides a "Cancel" button to stop ongoing processing. The `ProcessingHandler` implements logic to handle cancellation requests. This typically involves setting an internal flag and attempting to shut down the `ProcessPoolExecutor`. However, it's important to note that this does not immediately terminate worker processes that are already executing; it primarily prevents new tasks from starting and stops processing results from completed futures once the cancellation flag is checked.
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how data is handled and passed within the GUI and its interactions with background handlers reflect the current state and will require review and updates once the plan for these changes is finalized.
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness. The Unified Hierarchical View centralizes rule management in the GUI, and the `SourceRule` object serves as a clear data contract passed to the processing engine.

3
Presets/Poliigon.json
View File

@ -57,7 +57,8 @@
"target_type": "ROUGH",
"keywords": [
"GLOSS"
]
],
"is_gloss_source": true
},
{
"target_type": "AO",

BIN
gui/__pycache__/prediction_handler.cpython-313.pyc
View File

Binary file not shown.

33
gui/prediction_handler.py
View File

@ -114,6 +114,19 @@ def classify_files(file_list: List[str], config: Configuration) -> Dict[str, Lis
for compiled_regex, original_keyword, rule_index in patterns_list:
match = compiled_regex.search(filename)
if match:
# --- DEBUG LOG: Inspect available rule info ---
log.debug(f" Match found! Rule Index: {rule_index}, Original Keyword: '{original_keyword}', Target Type: '{target_type}'")
# Access the full rule details directly from the config's map_type_mapping list using the index
matched_rule_details = None
try:
matched_rule_details = config.map_type_mapping[rule_index] # Access rule by index
is_gloss_flag = matched_rule_details.get('is_gloss_source', False) # Get flag or default False
log.debug(f" Associated rule details: {matched_rule_details}")
log.debug(f" 'is_gloss_source' flag from rule: {is_gloss_flag}")
except IndexError:
log.warning(f" Could not access map_type_mapping rule at index {rule_index}. Cannot determine 'is_gloss_source' flag.")
is_gloss_flag = False # Default if rule cannot be accessed
# --- End DEBUG LOG ---
matched_item_type = target_type # The standard type (e.g., MAP_COL)
asset_name = None
# --- Asset Name Extraction Logic (Simplified Heuristic) ---
@ -129,7 +142,9 @@ def classify_files(file_list: List[str], config: Configuration) -> Dict[str, Lis
temp_grouped_files[asset_name].append({
'file_path': file_path_str,
'item_type': matched_item_type,
'asset_name': asset_name
'asset_name': asset_name,
# --- Store the flag retrieved from the rule ---
'is_gloss_source': is_gloss_flag # Store the boolean value obtained above
})
primary_asset_names.add(asset_name) # Mark this as a primary asset name
is_map = True
@ -310,7 +325,8 @@ class PredictionHandler(QObject):
# Create the single SourceRule for this input source
source_rule = SourceRule(
input_path=input_source_identifier, # Use the identifier provided
supplier_identifier=supplier_identifier # Set overridable field
supplier_identifier=supplier_identifier, # Set overridable field
preset_name=preset_name # Pass the selected preset name
)
log.debug(f"Created SourceRule for identifier: {input_source_identifier} with supplier: {supplier_identifier}")
@ -366,13 +382,26 @@ class PredictionHandler(QObject):
item_type_override = "FILE_IGNORE" # Fallback to FILE_IGNORE string
# Output format is determined by the engine, not predicted here. Leave as None.
output_format_override = None
# --- DEBUG LOG: Inspect data before FileRule creation ---
log.debug(f" Creating FileRule for: {file_info['file_path']}")
log.debug(f" Using item_type_override: {item_type_override}")
log.debug(f" Using target_asset_name_override: {target_asset_name_override}")
# Explicitly check and log the flag value from file_info
is_gloss_source_value = file_info.get('is_gloss_source', 'MISSING') # Get value or 'MISSING'
log.debug(f" Value for 'is_gloss_source' from file_info: {is_gloss_source_value}")
# --- End DEBUG LOG ---
# TODO: Need to verify FileRule constructor accepts is_gloss_source
# and pass is_gloss_source_value if it does.
# Pass the retrieved flag value to the constructor
file_rule = FileRule(
file_path=file_info['file_path'], # This is static info based on input
# --- Populate ONLY Overridable Fields ---
item_type_override=item_type_override,
target_asset_name_override=target_asset_name_override,
output_format_override=output_format_override,
is_gloss_source=is_gloss_source_value if isinstance(is_gloss_source_value, bool) else False, # Pass the flag, ensure boolean
# --- Leave Static Fields as Default/None ---
resolution_override=None,
channel_merge_instructions={},

87
processing_engine.py
View File

@ -326,7 +326,7 @@ class ProcessingEngine:
final_dir = output_base_path / supplier_sanitized / asset_name_sanitized
metadata_file_path = final_dir / self.config_obj.metadata_filename
if not overwrite and final_dir.exists() and metadata_file_path.is_file():
if not overwrite and final_dir.exists():
log.info(f"Output directory and metadata found for asset '{asset_name_sanitized}' and overwrite is False. Skipping.")
overall_status["skipped"].append(asset_name)
asset_skipped = True
@ -382,6 +382,8 @@ class ProcessingEngine:
# --- Organize Output ---
self._organize_output_files(
asset_rule=asset_rule,
workspace_path=workspace_path, # Pass the original workspace path
supplier_identifier=source_rule.supplier_identifier, # Pass supplier from SourceRule
output_base_path=output_base_path, # Pass output path
processed_maps_details_asset=processed_maps_details_asset,
merged_maps_details_asset=merged_maps_details_asset,
@ -513,20 +515,36 @@ class ProcessingEngine:
if img_prepared is None: raise ProcessingEngineError("Image data is None after MASK/Color prep.")
# Gloss -> Roughness Inversion (only if map_type starts with ROUGH and is_gloss_source is True)
if map_type.startswith('ROUGH') and is_gloss_source:
log.debug(f"Gloss Inversion Check: map_type='{map_type}', is_gloss_source={is_gloss_source}") # DEBUG ADDED
condition_met = map_type.startswith('ROUGH') and is_gloss_source # DEBUG ADDED
log.debug(f"Gloss Inversion Check: Condition met = {condition_met}") # DEBUG ADDED
if condition_met:
log.info(f"Performing Gloss->Roughness inversion for {source_path_abs.name}")
# Ensure grayscale before inversion
if len(img_prepared.shape) == 3:
log.debug("Gloss Inversion: Converting 3-channel image to grayscale before inversion.") # DEBUG ADDED
img_prepared = cv2.cvtColor(img_prepared, cv2.COLOR_RGB2GRAY) # Use RGB2GRAY as it should be RGB now
# Log stats *before* inversion (after potential grayscale conversion)
stats_before = _calculate_image_stats(img_prepared) # DEBUG ADDED
log.debug(f"Gloss Inversion: Image stats BEFORE inversion: {stats_before}") # DEBUG ADDED
# Normalize based on original source dtype before inversion
if source_dtype == np.uint16:
log.debug("Gloss Inversion: Normalizing uint16 data for inversion.") # DEBUG ADDED
img_float = 1.0 - (img_prepared.astype(np.float32) / 65535.0)
elif source_dtype == np.uint8:
log.debug("Gloss Inversion: Normalizing uint8 data for inversion.") # DEBUG ADDED
img_float = 1.0 - (img_prepared.astype(np.float32) / 255.0)
else: # Assuming float input is already 0-1 range
log.debug("Gloss Inversion: Assuming float data is already normalized for inversion.") # DEBUG ADDED
img_float = 1.0 - img_prepared.astype(np.float32)
img_prepared = np.clip(img_float, 0.0, 1.0) # Result is float32
# Log stats *after* inversion
stats_after = _calculate_image_stats(img_prepared) # DEBUG ADDED
log.debug(f"Gloss Inversion: Image stats AFTER inversion (float32): {stats_after}") # DEBUG ADDED
log.debug(f"Inverted gloss map stored as float32 for ROUGH, original dtype: {source_dtype}")
@ -882,12 +900,13 @@ class ProcessingEngine:
# Correction: _process_individual_maps receives the *engine's* temp_dir as workspace_path
source_path_abs = workspace_path / source_path_rel
map_type = file_rule.item_type_override # Use the explicit map type from the rule
# Determine if the source is gloss based on its type identifier and config
gloss_identifiers = getattr(self.config_obj, 'gloss_map_identifiers', [])
is_gloss_source = map_type in gloss_identifiers
# Determine if the source is gloss based on the flag set during prediction
# is_gloss_source = map_type in gloss_identifiers # <<< INCORRECT: Re-calculates based on target type
is_gloss_source = getattr(file_rule, 'is_gloss_source', False) # <<< CORRECT: Use flag from FileRule object
log.debug(f"Using is_gloss_source={is_gloss_source} directly from FileRule for {file_rule.file_path}") # DEBUG ADDED
original_extension = source_path_rel.suffix.lower() # Get from path
log.info(f"-- Asset '{asset_name}': Processing Individual Map: {map_type} (Source: {source_path_rel.name}) --")
log.info(f"-- Asset '{asset_name}': Processing Individual Map: {map_type} (Source: {source_path_rel.name}, IsGlossSource: {is_gloss_source}) --") # DEBUG: Added flag to log
current_map_details = {"derived_from_gloss": is_gloss_source}
source_bit_depth_found = None # Track if we've found the bit depth for this map type
@ -1260,8 +1279,8 @@ class ProcessingEngine:
# Start with the base metadata passed in (already contains name, category, archetype, stats, aspect, map_details)
final_metadata = current_asset_metadata.copy()
# Add supplier name from static config
final_metadata["supplier_name"] = self.config_obj.supplier_name
# Use the supplier identifier determined by rules/selection (from SourceRule)
final_metadata["supplier_name"] = source_rule.supplier_identifier or self.config_obj.supplier_name # Fallback to config if empty
# Populate map resolution details from processing results
final_metadata["processed_map_resolutions"] = {}
@ -1306,7 +1325,7 @@ class ProcessingEngine:
# Add processing info (using static config for preset name)
final_metadata["_processing_info"] = {
"preset_used": self.config_obj.preset_name, # Get from static config
"preset_used": getattr(source_rule, 'preset_name', self.config_obj.preset_name), # Use preset name from SourceRule (fallback to static config)
"timestamp_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"input_source": source_rule.supplier_identifier or "Unknown", # Use identifier from parent SourceRule
}
@ -1330,13 +1349,16 @@ class ProcessingEngine:
raise ProcessingEngineError(f"Failed to write metadata file {output_path} for asset '{asset_name}': {e}") from e
def _organize_output_files(self, asset_rule: AssetRule, output_base_path: Path, processed_maps_details_asset: Dict[str, Dict[str, Dict]], merged_maps_details_asset: Dict[str, Dict[str, Dict]], temp_metadata_path: Path):
def _organize_output_files(self, asset_rule: AssetRule, workspace_path: Path, supplier_identifier: str, output_base_path: Path, processed_maps_details_asset: Dict[str, Dict[str, Dict]], merged_maps_details_asset: Dict[str, Dict[str, Dict]], temp_metadata_path: Path):
"""
Moves/copies processed files for a specific asset from the engine's temp dir
to the final output structure, based on the AssetRule and static config.
and copies EXTRA files from the original workspace to the final output structure,
based on the AssetRule and static config.
Args:
asset_rule: The AssetRule object for this asset.
workspace_path: Path to the original workspace containing source files.
supplier_identifier: The supplier identifier from the SourceRule.
output_base_path: The final base output directory.
processed_maps_details_asset: Details of processed maps for this asset.
merged_maps_details_asset: Details of merged maps for this asset.
@ -1346,17 +1368,20 @@ class ProcessingEngine:
asset_name = asset_rule.asset_name
if not asset_name: raise ProcessingEngineError("Asset name missing for organization.")
# Get structure names from static config
supplier_name = self.config_obj.supplier_name
# Get structure names from static config and arguments
# supplier_name = self.config_obj.supplier_name # <<< ISSUE: Uses config supplier
metadata_filename = self.config_obj.metadata_filename
extra_subdir_name = self.config_obj.extra_files_subdir
if not supplier_name: raise ProcessingEngineError("Supplier name missing from config.")
# Use the supplier identifier passed from the SourceRule
if not supplier_identifier:
log.warning(f"Asset '{asset_name}': Supplier identifier missing in SourceRule. Using fallback 'UnknownSupplier'.")
supplier_identifier = "UnknownSupplier"
supplier_sanitized = _sanitize_filename(supplier_name)
supplier_sanitized = _sanitize_filename(supplier_identifier) # <<< FIX: Use passed identifier
asset_name_sanitized = _sanitize_filename(asset_name)
final_dir = output_base_path / supplier_sanitized / asset_name_sanitized
log.info(f"Organizing output files for asset '{asset_name_sanitized}' into: {final_dir}")
log.info(f"Organizing output files for asset '{asset_name_sanitized}' (Supplier: '{supplier_identifier}') into: {final_dir}")
try:
# Overwrite logic is handled in the main process() method before calling this
@ -1411,13 +1436,29 @@ class ProcessingEngine:
log.warning(f"Asset '{asset_name_sanitized}': Temporary metadata file path missing or file does not exist: {temp_metadata_path}")
# --- Handle Extra/Ignored/Unmatched Files (copy from original workspace) ---
# These also need copying from the original workspace_path.
# TODO: Revisit how these are handled. Should they be copied to temp first?
# For now, assume the caller handles copying these based on the SourceRule.
# log.warning("Extra/Ignored/Unmatched file organization not implemented in ProcessingEngine._organize_output_files yet.")
# Find relevant FileRules and copy from workspace_path. Needs workspace_path access.
# Let's assume the caller handles this for now.
# --- Handle "EXTRA" Files (copy from original workspace) ---
extra_dir = final_dir / extra_subdir_name
copied_extra_files = []
for file_rule in asset_rule.files:
if file_rule.item_type_override == "EXTRA":
try:
source_rel_path = Path(file_rule.file_path)
source_abs = workspace_path / source_rel_path
dest_abs = extra_dir / source_rel_path.name # Place in Extra subdir, keep original name
if source_abs.is_file():
log.debug(f"Asset '{asset_name_sanitized}': Copying EXTRA file: {source_abs.name} -> {extra_dir.relative_to(output_base_path)}/")
extra_dir.mkdir(parents=True, exist_ok=True)
shutil.copy2(str(source_abs), str(dest_abs)) # copy2 preserves metadata
copied_extra_files.append(source_rel_path.name)
else:
log.warning(f"Asset '{asset_name_sanitized}': Source file marked as EXTRA not found in workspace: {source_abs}")
except Exception as copy_err:
log.error(f"Asset '{asset_name_sanitized}': Failed copying EXTRA file '{file_rule.file_path}': {copy_err}", exc_info=True)
if copied_extra_files:
log.info(f"Asset '{asset_name_sanitized}': Copied {len(copied_extra_files)} EXTRA file(s) to '{extra_subdir_name}' subdirectory.")
log.info(f"Finished organizing output for asset '{asset_name_sanitized}'.")

2
rule_structure.py
View File

@ -9,6 +9,7 @@ class FileRule:
resolution_override: Tuple[int, int] = None
channel_merge_instructions: Dict[str, Any] = dataclasses.field(default_factory=dict)
output_format_override: str = None # Potentially others identified during integration
is_gloss_source: bool = False # Added flag to indicate if source is glossiness
def to_json(self) -> str:
return json.dumps(dataclasses.asdict(self), indent=4)
@ -43,6 +44,7 @@ class SourceRule:
high_level_sorting_parameters: Dict[str, Any] = dataclasses.field(default_factory=dict)
assets: List[AssetRule] = dataclasses.field(default_factory=list)
input_path: str = None
preset_name: str = None # Added to store the preset used for processing
def to_json(self) -> str:
return json.dumps(dataclasses.asdict(self), indent=4)