Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Initial Work on data-transfer refactor

Browse Source
This commit is contained in:
Rusfort
2025-04-29 23:11:44 +02:00
parent f66971f3fd
commit a6dc49ebb1
16 changed files with 1092 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
Documentation/00_Overview.md
View File

@@ -12,7 +12,7 @@ This documentation strictly excludes details on environment setup, dependency in
## Architecture and Codebase Summary
For developers interested in contributing, the tool's architecture is designed around a **Core Processing Engine** (`asset_processor.py`) that handles the pipeline for single assets, supported by a **Configuration System** (`configuration.py` and `config.py` with `Presets/*.json`). Multiple interfaces are provided: a **Graphical User Interface** (`gui/`), a **Command-Line Interface** (`main.py`), and a **Directory Monitor** (`monitor.py`). Optional **Blender Integration** (`blenderscripts/`) is also included.
For developers interested in contributing, the tool's architecture is designed around a **Core Processing Engine** (`asset_processor.py`) that handles the pipeline for single assets. This engine is supported by a **Configuration System** (`configuration.py` and `config.py` with `Presets/*.json`) and a new **Hierarchical Rule System** (`rule_structure.py`) that allows dynamic overrides of static configurations at Source, Asset, and File levels. Multiple interfaces are provided: a **Graphical User Interface** (`gui/`), a **Command-Line Interface** (`main.py`), and a **Directory Monitor** (`monitor.py`). Optional **Blender Integration** (`blenderscripts/`) is also included. Key new files supporting the hierarchical rule system include `rule_structure.py`, `gui/rule_hierarchy_model.py`, and `gui/rule_editor_widget.py`.
The codebase is organized into key directories and files reflecting these components. The `gui/` directory contains all GUI-related code, `Presets/` holds configuration presets, and `blenderscripts/` contains scripts for Blender interaction. The core logic resides in files like `asset_processor.py`, `configuration.py`, `config.py`, `main.py`, and `monitor.py`. The processing pipeline involves steps such as file classification, map processing, channel merging, and metadata generation.

10
Documentation/02_Developer_Guide/01_Architecture.md
View File

@@ -15,6 +15,16 @@ The Asset Processor Tool is designed to process 3D asset source files into a sta
These interfaces exchange data structures containing information about each file and asset set being processed.
4. **Optional Integration:** Includes scripts and logic for integrating with external software, specifically Blender, to automate material and node group creation.
## Hierarchical Rule System
A key addition to the architecture is the **Hierarchical Rule System**, which provides a dynamic layer of configuration that can override the static settings loaded from presets. This system is represented by a hierarchy of rule objects: `SourceRule`, `AssetRule`, and `FileRule`.
* **SourceRule:** Represents rules applied at the top level, typically corresponding to an entire input source (e.g., a ZIP file or folder).
* **AssetRule:** Represents rules applied to a specific asset within a source (a source can contain multiple assets).
* **FileRule:** Represents rules applied to individual files within an asset.
This hierarchy allows for fine-grained control over processing parameters. When the processing engine needs a configuration value (e.g., a naming convention or image processing setting), it first checks the `FileRule` for the specific file, then the `AssetRule` for the asset containing the file, then the `SourceRule` for the overall source. If a value is not found at any of these dynamic levels, it falls back to the static configuration loaded from the selected preset. This prioritization logic (File > Asset > Source > Static Preset) ensures that the most specific rule available is always used.
## Core Components
* `config.py`: Defines core, global settings and constants.

27
Documentation/02_Developer_Guide/03_Key_Components.md
View File

@@ -14,7 +14,32 @@ The `AssetProcessor` class is the central engine of the tool. It is responsible
* Merging channels from different maps according to merge rules.
* Generating the `metadata.json` file containing details about the processed asset.
* Organizing the final output files into the structured library directory.
* Providing methods (`get_detailed_file_predictions`) used by the GUI for previewing file classification.
* Accepting a `SourceRule` object which represents the hierarchical rules (Source, Asset, File) that can override static configuration values.
* Implementing the logic (`_get_rule_with_fallback`) to retrieve configuration values by prioritizing rules in the order: File -> Asset -> Source -> Static Config.
* Applying this hierarchical rule logic during file classification, prediction, image processing, and metadata generation.
* Providing methods (`get_detailed_file_predictions`) used by the GUI for previewing file classification, now incorporating the hierarchical rules.
## `Rule Structure` (`rule_structure.py`)
This module defines the data structures used to represent the hierarchical processing rules:
* `SourceRule`: A dataclass representing rules applied at the source level. It contains nested `AssetRule` objects.
* `AssetRule`: A dataclass representing rules applied at the asset level. It contains nested `FileRule` objects.
* `FileRule`: A dataclass representing rules applied at the file level.
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `map_type_override`) and support serialization (Pickle, JSON) to allow them to be passed between different parts of the application, including across process boundaries.
## `Rule Hierarchy Model` (`gui/rule_hierarchy_model.py`)
The `RuleHierarchyModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
* Wrap a `SourceRule` object and expose its hierarchical structure (Source -> Asset -> File) to a `QTreeView`.
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy.
* Enable navigation and selection within the rule hierarchy in the GUI.
## `Rule Editor Widget` (`gui/rule_editor_widget.py`)
The `RuleEditorWidget` is a custom Qt widget that provides a user interface for viewing and editing the attributes of a selected rule object (`SourceRule`, `AssetRule`, or `FileRule`). It dynamically generates input fields (e.g., line edits, checkboxes) based on the attributes of the rule object it is currently displaying. It allows users to modify the dynamic rule parameters through the GUI.
## `Configuration` (`configuration.py`)

11
Documentation/02_Developer_Guide/05_Processing_Pipeline.md
View File

@@ -2,7 +2,9 @@
This document details the step-by-step technical process executed by the `AssetProcessor` class (`asset_processor.py`) when processing a single asset.
The `AssetProcessor.process()` method orchestrates the following pipeline:
The `AssetProcessor.process()` method orchestrates the following pipeline. A key aspect of this pipeline is the integration of the **Hierarchical Rule System**, which allows dynamic rules (Source, Asset, File) to override static configuration values loaded from presets. The `_get_rule_with_fallback` helper method is used throughout the pipeline to retrieve configuration values, prioritizing rules in the order: File -> Asset -> Source -> Static Config.
The pipeline steps are:
1. **Workspace Setup (`_setup_workspace`)**:
* Creates a temporary directory using `tempfile.mkdtemp()` to isolate the processing of the current asset.
@@ -14,7 +16,7 @@ The `AssetProcessor.process()` method orchestrates the following pipeline:
3. **File Inventory and Classification (`_inventory_and_classify_files`)**:
* Scans the contents of the temporary workspace.
* Uses the pre-compiled regex patterns from the loaded `Configuration` object to classify each file.
* Uses the pre-compiled regex patterns from the loaded `Configuration` object and the rules from the `SourceRule` object to classify each file. The hierarchical rules can influence classification patterns and other parameters used in this step.
* Classification follows a multi-pass approach for priority:
* Explicitly marked `Extra/` files (using `move_to_extra_patterns` regex).
* Model files (using `model_patterns` regex).
@@ -27,6 +29,7 @@ The `AssetProcessor.process()` method orchestrates the following pipeline:
* Sorts potential map variants based on preset rule order, keyword order within the rule, and finally alphabetical path to determine suffix assignment priority (`-1`, `-2`, etc.).
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
* This step now utilizes the hierarchical rules from the `SourceRule` object. For example, the `supplier_identifier`, `asset_type`, and `asset_name_override` attributes from the rules can override the values determined from the static configuration or input file names. The `_get_rule_with_fallback` method is used here to apply the rule prioritization logic.
* Determines the base asset name using `source_naming_convention` rules from the `Configuration` (separators, indices), with fallbacks to common prefixes or the input name. Handles multiple distinct assets within a single input source.
* Determines the asset category (`Texture`, `Asset`, `Decal`) based on the presence of model files or `decal_keywords` in the `Configuration`.
* Determines the asset archetype (e.g., `Wood`, `Metal`) by matching keywords from `archetype_rules` (in `Configuration`) against file stems or the determined base name.
@@ -37,7 +40,7 @@ The `AssetProcessor.process()` method orchestrates the following pipeline:
* If both exist, processing for this specific asset is skipped, marked as "skipped", and the pipeline moves to the next asset (if processing multiple assets from one source) or finishes.
6. **Map Processing (`_process_maps`)**:
* Iterates through the files classified as texture maps for the current asset.
* Iterates through the files classified as texture maps for the current asset. Configuration values used in this step, such as target resolutions, bit depth rules, and output format rules, are now retrieved using the `_get_rule_with_fallback` method, allowing them to be overridden by the hierarchical rules.
* Loads the image using `cv2.imread` (handling grayscale and unchanged flags). Converts BGR to RGB internally for consistency (except for saving non-EXR formats).
* Handles Glossiness-to-Roughness inversion if necessary (loads gloss, inverts `1.0 - img/norm`, prioritizes gloss source if both exist).
* Resizes the image to target resolutions defined in `IMAGE_RESOULTIONS` (from `Configuration`) using `cv2.resize` (`INTER_LANCZOS4` for downscaling). Upscaling is generally avoided by checks.
@@ -53,6 +56,7 @@ The `AssetProcessor.process()` method orchestrates the following pipeline:
* Stores details about each processed map (path, resolution, format, stats, etc.) in `processed_maps_details_asset`.
7. **Map Merging (`_merge_maps_from_source`)**:
* The `MAP_MERGE_RULES` themselves can be influenced by the hierarchical rules. Additionally, parameters within the merge rules, such as default channel values or output format/bit depth, are retrieved using `_get_rule_with_fallback`, allowing dynamic overrides.
* Iterates through the `MAP_MERGE_RULES` defined in the `Configuration`.
* Identifies the required *source* map files needed as input for each merge rule based on the classified files.
* Determines common resolutions available across the required input maps.
@@ -65,6 +69,7 @@ The `AssetProcessor.process()` method orchestrates the following pipeline:
* Stores details about each merged map in `merged_maps_details_asset`.
8. **Metadata File Generation (`_generate_metadata_file`)**:
* The metadata generated now includes information derived from the applied hierarchical rules, in addition to the static configuration and processing results.
* Collects all determined information for the current asset: base metadata, details from `processed_maps_details_asset` and `merged_maps_details_asset`, list of ignored files, source preset used, etc.
* Writes this collected data into the `metadata.json` file within the temporary workspace using `json.dump`.

15
Documentation/02_Developer_Guide/06_GUI_Internals.md
View File

@@ -16,12 +16,14 @@ The `MainWindow` class is the central component of the GUI application. It is re
* Connecting user interactions (button clicks, drag-and-drop events, checkbox states, spinbox values) to corresponding methods (slots) within the `MainWindow` or other handler classes.
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
* Integrating the `QTreeView` (displaying the rule hierarchy via `RuleHierarchyModel`) and the `RuleEditorWidget` for interactive rule editing.
* Connecting the selection changes in the `QTreeView` to update the `RuleEditorWidget` with the selected rule object's attributes.
## Threading and Background Tasks
To keep the UI responsive during intensive operations like asset processing and file preview generation, the GUI utilizes background threads managed by `QThread`.
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool and monitors their completion. It uses Qt signals to communicate progress updates, file status changes, and overall processing completion back to the `MainWindow` on the main UI thread. It also handles the execution of optional Blender scripts via subprocess calls after processing. This handler processes and utilizes data structures received from the core processing engine, such as status summaries.
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool and monitors their completion. It uses Qt signals to communicate progress updates, file status changes, and overall processing completion back to the `MainWindow` on the main UI thread. It also handles the execution of optional Blender scripts via subprocess calls after processing. This handler processes and utilizes data structures received from the core processing engine, such as status summaries. It receives the initial `SourceRule` object generated by the `PredictionHandler` and passes it to the `AssetProcessor` for processing.
* **`PredictionHandler` (`gui/prediction_handler.py`):** This class also runs in a separate `QThread`. It is responsible for generating the detailed file classification previews displayed in the preview table. It calls methods on the `AssetProcessor` (`get_detailed_file_predictions`) to perform the analysis in the background. It uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks. Results are sent back to the `MainWindow` via Qt signals to update the preview table data. This handler works with data structures containing file prediction details.
## Communication (Signals and Slots)
@@ -54,6 +56,17 @@ The `data()` method of the `PreviewTableModel` then accesses this structured `se
The `PreviewSortFilterProxyModel` operates on this structured data, implementing a multi-level sort based on source asset, row type (main vs. additional-only), and file paths within those types.
* Calls methods on the `AssetProcessor` (specifically `get_detailed_file_predictions`) to perform the analysis in the background. It uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks. Results are sent back to the `MainWindow` via Qt signals to update the preview table data. This handler works with data structures containing file prediction details. It is also responsible for generating the initial `SourceRule` hierarchy based on the input files and the selected preset, and emitting a signal to provide this `SourceRule` object to the `MainWindow`.
## Rule Hierarchy UI (`gui/rule_hierarchy_model.py`, `gui/rule_editor_widget.py`)
The GUI now includes dedicated components for visualizing and editing the hierarchical processing rules:
* **`Rule Hierarchy Model` (`gui/rule_hierarchy_model.py`):** This class implements `QAbstractItemModel` to expose the structure of a `SourceRule` object (Source -> Asset -> File) to a `QTreeView`. It allows the `QTreeView` to display the hierarchy and enables user selection of individual rule objects (SourceRule, AssetRule, or FileRule).
* **`Rule Editor Widget` (`gui/rule_editor_widget.py`):** This custom widget provides a dynamic form for editing the attributes of the currently selected rule object from the hierarchy tree. When a rule object is selected in the `QTreeView`, the `MainWindow` updates the `RuleEditorWidget` to display and allow modification of that object's specific parameters.
These components are integrated into the `MainWindow`. The `PredictionHandler` generates the initial `SourceRule` hierarchy, which is then set on the `RuleHierarchyModel`. The `QTreeView` displays this model, and selection changes in the tree trigger updates to the `RuleEditorWidget`, allowing users to interactively modify the dynamic rules before processing. Edits made in the `RuleEditorWidget` directly modify the attributes of the underlying rule objects in the `SourceRule` hierarchy.
### Preview Table Column Configuration
The display and behavior of the columns in the `QTableView` are configured in `gui/main_window.py`. The current configuration is as follows: