Pre-Codebase-review commit :3
Codebase dedublication and Cleanup refactor Documentation updated as well Preferences update Removed testfiles from repository
This commit is contained in:
@@ -12,9 +12,9 @@ This documentation strictly excludes details on environment setup, dependency in
|
||||
|
||||
## Architecture and Codebase Summary
|
||||
|
||||
For developers interested in contributing, the tool's architecture is designed around a **Core Processing Engine** (`asset_processor.py`) that handles the pipeline for single assets. This engine is supported by a **Configuration System** (`configuration.py` and `config.py` with `Presets/*.json`) and a new **Hierarchical Rule System** (`rule_structure.py`) that allows dynamic overrides of static configurations at Source, Asset, and File levels. Multiple interfaces are provided: a **Graphical User Interface** (`gui/`), a **Command-Line Interface** (`main.py`), and a **Directory Monitor** (`monitor.py`). Optional **Blender Integration** (`blenderscripts/`) is also included. Key new files supporting the hierarchical rule system include `rule_structure.py`, `gui/rule_hierarchy_model.py`, and `gui/rule_editor_widget.py`.
|
||||
For developers interested in contributing, the tool's architecture centers on a **Core Processing Engine** (`processing_engine.py`) executing a pipeline based on a **Hierarchical Rule System** (`rule_structure.py`) and a **Configuration System** (`configuration.py` loading `config/app_settings.json` and `Presets/*.json`). The **Graphical User Interface** (`gui/`) has been significantly refactored: `MainWindow` (`main_window.py`) acts as a coordinator, delegating tasks to specialized widgets (`MainPanelWidget`, `PresetEditorWidget`, `LogConsoleWidget`) and background handlers (`RuleBasedPredictionHandler`, `LLMPredictionHandler`, `LLMInteractionHandler`, `AssetRestructureHandler`). The **Directory Monitor** (`monitor.py`) now processes archives asynchronously using a thread pool and utility functions (`utils/prediction_utils.py`, `utils/workspace_utils.py`). The **Command-Line Interface** entry point (`main.py`) primarily launches the GUI, with core CLI functionality currently non-operational. Optional **Blender Integration** (`blenderscripts/`) remains. A new `utils/` directory houses shared helper functions.
|
||||
|
||||
The codebase is organized into key directories and files reflecting these components. The `gui/` directory contains all GUI-related code, `Presets/` holds configuration presets, and `blenderscripts/` contains scripts for Blender interaction. The core logic resides in files like `asset_processor.py`, `configuration.py`, `config.py`, `main.py`, and `monitor.py`. The processing pipeline involves steps such as file classification, map processing, channel merging, and metadata generation.
|
||||
The codebase reflects this structure. The `gui/` directory contains the refactored UI components, `utils/` holds shared utilities, `Presets/` contains JSON presets, and `blenderscripts/` holds Blender scripts. Core logic resides in `processing_engine.py`, `configuration.py`, `rule_structure.py`, `monitor.py`, and `main.py`. The processing pipeline, executed by `processing_engine.py`, relies entirely on the input `SourceRule` and static configuration for steps like map processing, channel merging, and metadata generation.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
|
||||
@@ -21,16 +21,18 @@ python -m gui.main_window
|
||||
* **Preset Selector:** Choose the preset to use for *processing* the current queue. This dropdown now includes a new option: "- LLM Interpretation -". Selecting this option will use the experimental LLM Predictor instead of the traditional rule-based prediction system defined in presets.
|
||||
* **Output Directory:** Set the output path (defaults to `config/app_settings.json`, use "Browse...")
|
||||
* **Drag and Drop Area:** Add asset `.zip`, `.rar`, `.7z` files, or folders by dragging and dropping them here.
|
||||
* **Preview Table:** Shows queued assets in a hierarchical view (Source -> Asset -> File). Initially, this area displays a message prompting you to select a preset. Once a preset is selected from the Preset List, the detailed file preview will load here. The mode of the preview depends on the "View" menu:
|
||||
* **Detailed Preview (Default):** Lists all files, predicted status (`Mapped`, `Model`, `Extra`, `Unrecognised`, `Ignored`, `Error`), output name, etc., based on the selected *processing* preset. The columns displayed are: Name, Target Asset, Supplier, Asset Type, Item Type. The "Target Asset" column stretches to fill available space, while others resize to content. The previous "Status" and "Output Path" columns have been removed. Text colors are applied to cells based on the status of the individual file they represent. Rows use alternating background colors per asset group for visual separation.
|
||||
* **Simple View (Preview Disabled):** Lists only top-level input asset paths.
|
||||
* **Preview Table:** Shows queued assets in a hierarchical view (Source -> Asset -> File). Assets (files, directories, archives) added via drag-and-drop appear immediately in the table.
|
||||
* If no preset is selected ("-- Select a Preset --"), added items (including files within directories/archives) are displayed with empty prediction fields (Target Asset, Asset Type, Item Type), which can be manually edited.
|
||||
* If a valid preset or LLM mode is selected, the table populates with prediction results as they become available.
|
||||
* The table always displays the detailed view structure with columns: Name, Target Asset, Supplier, Asset Type, Item Type. The "Target Asset" column stretches to fill available space.
|
||||
* **Coloring:** The *text color* of file items is determined by their Item Type (colors defined in `config/app_settings.json`). The *background color* of file items is a 30% darker shade of their parent asset's background, helping to visually group files within an asset. Asset rows themselves may use alternating background colors based on the application theme.
|
||||
* **Progress Bar:** Shows overall processing progress.
|
||||
* **Blender Post-Processing:** Checkbox to enable Blender scripts. If enabled, shows fields and browse buttons for target `.blend` files (defaults from `config/app_settings.json`).
|
||||
* **Options & Controls (Bottom):**
|
||||
* `Overwrite Existing`: Checkbox to force reprocessing.
|
||||
* `Workers`: Spinbox for concurrent processes.
|
||||
* `Clear Queue`: Button to clear the queue and preview.
|
||||
* `Start Processing`: Button to start processing the queue. This button is disabled until a valid preset is selected from the Preset List.
|
||||
* `Start Processing`: Button to start processing the queue. This button is enabled as long as there are items listed in the Preview Table. When clicked, any items that do not have a value assigned in the "Target Asset" column will be automatically ignored for that processing run.
|
||||
* `Cancel`: Button to attempt stopping processing.
|
||||
* **Re-interpret Selected with LLM:** This button appears when the "- LLM Interpretation -" preset is selected. It allows you to re-process only the currently selected items in the Preview Table using the LLM, without affecting other items in the queue. This is useful for refining predictions on specific assets.
|
||||
* **Status Bar:** Displays current status, errors, and completion messages. During LLM processing, the status bar will show messages indicating the progress of the LLM requests.
|
||||
|
||||
@@ -6,17 +6,17 @@ This document provides a high-level overview of the Asset Processor Tool's archi
|
||||
|
||||
The Asset Processor Tool is designed to process 3D asset source files into a standardized library format. Its high-level architecture consists of:
|
||||
|
||||
1. **Core Processing Engine (`processing_engine.py`):** The primary component responsible for executing the asset processing pipeline for a single input asset based on a provided `SourceRule` object and static configuration. The older `asset_processor.py` remains in the codebase for reference but is no longer used in the main processing flow.
|
||||
2. **Prediction System:** Responsible for analyzing input files and generating the initial `SourceRule` hierarchy with predicted values. This system now includes two alternative components:
|
||||
* **Rule-Based Predictor (`prediction_handler.py`):** Uses predefined rules from presets to classify files and determine initial processing parameters.
|
||||
* **LLM Predictor (`gui/llm_prediction_handler.py`):** An experimental alternative that uses a Large Language Model (LLM) to interpret file contents and context to predict processing parameters. Its role is to generate `SourceRule` objects based on LLM output, which are then used by the processing pipeline.
|
||||
1. **Core Processing Engine (`processing_engine.py`):** The primary component responsible for executing the asset processing pipeline for a single input asset based on a provided `SourceRule` object and static configuration. The previous `asset_processor.py` has been removed.
|
||||
2. **Prediction System:** Responsible for analyzing input files and generating the initial `SourceRule` hierarchy with predicted values. This system utilizes a base handler (`gui/base_prediction_handler.py::BasePredictionHandler`) with specific implementations:
|
||||
* **Rule-Based Predictor (`gui/prediction_handler.py::RuleBasedPredictionHandler`):** Uses predefined rules from presets to classify files and determine initial processing parameters.
|
||||
* **LLM Predictor (`gui/llm_prediction_handler.py::LLMPredictionHandler`):** An experimental alternative that uses a Large Language Model (LLM) to interpret file contents and context to predict processing parameters.
|
||||
3. **Configuration System (`Configuration`):** Handles loading core settings (including centralized type definitions and LLM-specific configuration) and merging them with supplier-specific rules defined in JSON presets and the persistent `config/suppliers.json` file.
|
||||
4. **Multiple Interfaces:** Provides different ways to interact with the tool:
|
||||
* Graphical User Interface (GUI)
|
||||
* Command-Line Interface (CLI)
|
||||
* Command-Line Interface (CLI) - *Note: The primary CLI execution logic (`run_cli` in `main.py`) is currently non-functional/commented out post-refactoring.*
|
||||
* Directory Monitor for automated processing.
|
||||
The GUI now acts as the primary source of truth for processing rules, generating and managing the `SourceRule` hierarchy before sending it to the processing engine. It also accumulates prediction results from multiple input sources before updating the view. The CLI and Monitor interfaces can also generate `SourceRule` objects to bypass the GUI for automated workflows.
|
||||
5. **Optional Integration:** Includes scripts and logic for integrating with external software, specifically Blender, to automate material and node group creation.
|
||||
The GUI acts as the primary source of truth for processing rules, coordinating the generation and management of the `SourceRule` hierarchy before sending it to the processing engine. It accumulates prediction results from multiple input sources before updating the view. The Monitor interface can also generate `SourceRule` objects (using `utils/prediction_utils.py`) to bypass the GUI for automated workflows.
|
||||
5. **Optional Integration:** Includes scripts (`blenderscripts/`) for integrating with Blender. Logic for executing these scripts was intended to be centralized in `utils/blender_utils.py`, but this utility has not yet been implemented.
|
||||
|
||||
## Hierarchical Rule System
|
||||
|
||||
@@ -35,22 +35,33 @@ This hierarchy allows for fine-grained control over processing parameters. The G
|
||||
* `Presets/*.json`: Supplier-specific JSON files defining rules for file interpretation and initial prediction.
|
||||
* `configuration.py` (`Configuration` class): Loads `config/app_settings.json` settings and merges them with a selected preset, pre-compiling regex patterns for efficiency. This static configuration is used by the processing engine.
|
||||
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses used to represent the hierarchical processing rules.
|
||||
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The GUI is responsible for generating and managing the `SourceRule` hierarchy via the Unified View, accumulating prediction results, and interacting with background handlers (`ProcessingHandler`, `PredictionHandler`).
|
||||
* `unified_view_model.py`: Implements the `QAbstractItemModel` for the Unified Hierarchical View, holding the `SourceRule` data, handling inline editing (including direct model restructuring for `target_asset_name_override`), and managing row coloring based on config definitions.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations for inline editing in the Unified View, including the new `SupplierSearchDelegate` for supplier name auto-completion and management.
|
||||
* `prediction_handler.py`: Generates the initial `SourceRule` hierarchy with predicted values for a single input source based on its files and the selected preset. It uses the `"standard_type"` from the configuration's `FILE_TYPE_DEFINITIONS` to populate `FileRule.standard_map_type` and implements a two-pass classification logic to handle and prioritize bit-depth variants (e.g., `_DISP16_` vs `_DISP_`).
|
||||
* `processing_engine.py` (`ProcessingEngine` class): The new core component that executes the processing pipeline for a single `SourceRule` object using the static `Configuration`. A new instance is created per task for state isolation. It contains no internal prediction or fallback logic. Supplier overrides from the GUI are correctly preserved and used by the engine for output path generation and metadata.
|
||||
* `asset_processor.py` (`AssetProcessor` class): The older processing engine, kept for reference but not used in the main processing flow.
|
||||
* `main.py`: The entry point for the Command-Line Interface (CLI). It handles argument parsing, logging, parallel processing orchestration, and triggering Blender scripts. It now orchestrates processing by passing `SourceRule` objects to the `ProcessingEngine`.
|
||||
* `monitor.py`: Implements the directory monitoring feature using `watchdog`.
|
||||
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The `MainWindow` (`main_window.py`) acts as a coordinator, orchestrating interactions between various components. Key GUI components include:
|
||||
* `main_panel_widget.py::MainPanelWidget`: Contains the primary controls for loading sources, selecting presets, viewing/editing rules, and initiating processing.
|
||||
* `preset_editor_widget.py::PresetEditorWidget`: Provides the interface for managing presets.
|
||||
* `log_console_widget.py::LogConsoleWidget`: Displays application logs.
|
||||
* `unified_view_model.py::UnifiedViewModel`: Implements the `QAbstractItemModel` for the hierarchical rule view, holding `SourceRule` data and managing display logic (coloring, etc.). Caches configuration data for performance.
|
||||
* `rule_hierarchy_model.py::RuleHierarchyModel`: A simpler model used internally by the `UnifiedViewModel` to manage the `SourceRule` data structure.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations for inline editing in the rule view.
|
||||
* `asset_restructure_handler.py::AssetRestructureHandler`: Handles complex model updates when a file's target asset is changed via the GUI, ensuring the `SourceRule` hierarchy is correctly modified.
|
||||
* `base_prediction_handler.py::BasePredictionHandler`: Abstract base class for prediction logic.
|
||||
* `prediction_handler.py::RuleBasedPredictionHandler`: Generates the initial `SourceRule` hierarchy based on presets and file analysis. Inherits from `BasePredictionHandler`.
|
||||
* `llm_prediction_handler.py::LLMPredictionHandler`: Experimental predictor using an LLM. Inherits from `BasePredictionHandler`.
|
||||
* `llm_interaction_handler.py::LLMInteractionHandler`: Manages communication with the LLM service for the LLM predictor.
|
||||
* `processing_engine.py` (`ProcessingEngine` class): The core component that executes the processing pipeline for a single `SourceRule` object using the static `Configuration`. A new instance is created per task for state isolation.
|
||||
* `main.py`: The main entry point for the application. Primarily launches the GUI. Contains commented-out/non-functional CLI logic (`run_cli`).
|
||||
* `monitor.py`: Implements the directory monitoring feature using `watchdog`. It now processes archives asynchronously using a `ThreadPoolExecutor`, leveraging `utils.prediction_utils.py` for rule generation and `utils.workspace_utils.py` for workspace management before invoking the `ProcessingEngine`.
|
||||
* `blenderscripts/`: Contains Python scripts designed to be executed *within* Blender for post-processing tasks.
|
||||
* `utils/`: Directory containing utility modules:
|
||||
* `workspace_utils.py`: Contains functions like `prepare_processing_workspace` for handling temporary directories and archive extraction.
|
||||
* `prediction_utils.py`: Contains functions like `generate_source_rule_from_archive` used by the monitor for rule-based prediction.
|
||||
* `blender_utils.py`: (Intended location for Blender script execution logic, currently not implemented).
|
||||
|
||||
## Processing Pipeline (Simplified)
|
||||
|
||||
The primary processing engine (`processing_engine.py`) executes a series of steps for each asset based on the provided `SourceRule` object and static configuration:
|
||||
|
||||
1. Extraction of input to a temporary workspace.
|
||||
2. Classification of files (map, model, extra, ignored, unrecognised) using preset rules.
|
||||
1. Extraction of input to a temporary workspace (using `utils.workspace_utils.py`).
|
||||
2. Classification of files (map, model, extra, ignored, unrecognised) based *only* on the provided `SourceRule` object (classification/prediction happens *before* the engine is called).
|
||||
3. Determination of base metadata (asset name, category, archetype).
|
||||
4. Skip check if output exists and overwrite is not forced.
|
||||
5. Processing of maps (resize, format/bit depth conversion, inversion, stats calculation).
|
||||
@@ -58,6 +69,6 @@ The primary processing engine (`processing_engine.py`) executes a series of step
|
||||
7. Generation of `metadata.json` file.
|
||||
8. Organization of processed files into the final output structure.
|
||||
9. Cleanup of the temporary workspace.
|
||||
10. (Optional) Execution of Blender scripts for post-processing.
|
||||
10. (Optional) Execution of Blender scripts (currently triggered directly, intended to use `utils.blender_utils.py`).
|
||||
|
||||
This architecture allows for a modular design, separating configuration, rule generation/management (primarily in the GUI), and core processing execution. The `SourceRule` object serves as a clear data contract between the GUI/prediction layer and the processing engine. Parallel processing is utilized for efficiency, and background threads keep the GUI responsive.
|
||||
This architecture allows for a modular design, separating configuration, rule generation/management (GUI, Monitor utilities), and core processing execution. The `SourceRule` object serves as a clear data contract between the rule generation layer and the processing engine. Parallel processing (in Monitor) and background threads (in GUI) are utilized for efficiency and responsiveness.
|
||||
@@ -4,69 +4,90 @@ This document outlines the key files and directories within the Asset Processor
|
||||
|
||||
```
|
||||
Asset_processor_tool/
|
||||
├── asset_processor.py # Older core class, kept for reference (not used in main flow)
|
||||
├── config.py # Core settings, constants, and definitions for allowed asset/file types
|
||||
├── config/ # Directory for configuration files
|
||||
│ └── suppliers.json # Persistent list of known supplier names for GUI auto-completion
|
||||
├── configuration.py # Class for loading and accessing configuration (merges config.py and presets)
|
||||
├── detailed_documentation_plan.md # (Existing file, potentially outdated)
|
||||
├── configuration.py # Class for loading and accessing configuration (merges app_settings.json and presets)
|
||||
├── Dockerfile # Instructions for building the Docker container image
|
||||
├── documentation_plan.md # Plan for the new documentation structure (this plan)
|
||||
├── documentation.txt # Original developer documentation (to be migrated)
|
||||
├── main.py # CLI Entry Point & processing orchestrator (calls processing_engine)
|
||||
├── monitor.py # Directory monitoring script for automated processing
|
||||
├── processing_engine.py # New core class handling single asset processing based on SourceRule
|
||||
├── readme.md # Original main documentation file (to be migrated)
|
||||
├── readme.md.bak # Backup of readme.md
|
||||
├── main.py # Main application entry point (primarily GUI launcher)
|
||||
├── monitor.py # Directory monitoring script for automated processing (async)
|
||||
├── processing_engine.py # Core class handling single asset processing based on SourceRule
|
||||
├── requirements-docker.txt # Dependencies specifically for the Docker environment
|
||||
├── requirements.txt # Python package dependencies for standard execution
|
||||
├── rule_structure.py # Dataclasses for hierarchical rules (SourceRule, AssetRule, FileRule)
|
||||
├── blenderscripts/ # Scripts for integration with Blender
|
||||
│ ├── create_materials.py # Script to create materials linking to node groups
|
||||
│ └── create_nodegroups.py # Script to create node groups from processed assets
|
||||
├── Deprecated-POC/ # Directory containing original proof of concept scripts
|
||||
│ ├── Blender-MaterialsFromNodegroups.py
|
||||
│ ├── Blender-NodegroupsFromPBRSETS.py
|
||||
│ └── Standalonebatcher-Main.py
|
||||
├── Documentation/ # New directory for organized documentation (this structure)
|
||||
├── config/ # Directory for configuration files
|
||||
│ ├── app_settings.json # Core settings, constants, and type definitions
|
||||
│ └── suppliers.json # Persistent list of known supplier names for GUI auto-completion
|
||||
├── Deprecated/ # Contains old code, documentation, and POC scripts
|
||||
│ ├── ...
|
||||
├── Documentation/ # Directory for organized documentation (this structure)
|
||||
│ ├── 00_Overview.md
|
||||
│ ├── 01_User_Guide/
|
||||
│ └── 02_Developer_Guide/
|
||||
├── gui/ # Contains files related to the Graphical User Interface
|
||||
│ ├── delegates.py # Custom delegates for inline editing in Unified View
|
||||
│ ├── main_window.py # Main GUI application window and layout
|
||||
│ ├── processing_handler.py # Handles background processing logic for the GUI
|
||||
│ ├── prediction_handler.py # Generates initial SourceRule hierarchy with predictions
|
||||
│ ├── unified_view_model.py # Model for the Unified Hierarchical View
|
||||
│ └── ... # Other GUI components
|
||||
├── Presets/ # Preset definition files
|
||||
├── gui/ # Contains files related to the Graphical User Interface (PySide6)
|
||||
│ ├── asset_restructure_handler.py # Handles model updates for target asset changes
|
||||
│ ├── base_prediction_handler.py # Abstract base class for prediction logic
|
||||
│ ├── config_editor_dialog.py # Dialog for editing configuration files
|
||||
│ ├── delegates.py # Custom delegates for inline editing in rule view
|
||||
│ ├── llm_interaction_handler.py # Manages communication with LLM service
|
||||
│ ├── llm_prediction_handler.py # LLM-based prediction handler
|
||||
│ ├── log_console_widget.py # Widget for displaying logs
|
||||
│ ├── main_panel_widget.py # Main panel containing core GUI controls
|
||||
│ ├── main_window.py # Main GUI application window (coordinator)
|
||||
│ ├── prediction_handler.py # Rule-based prediction handler
|
||||
│ ├── preset_editor_widget.py # Widget for managing presets
|
||||
│ ├── preview_table_model.py # Model for the (deprecated?) preview table
|
||||
│ ├── rule_editor_widget.py # Widget containing the rule hierarchy view and editor
|
||||
│ ├── rule_hierarchy_model.py # Internal model for rule hierarchy data
|
||||
│ └── unified_view_model.py # QAbstractItemModel for the rule hierarchy view
|
||||
├── llm_prototype/ # Files related to the experimental LLM predictor prototype
|
||||
│ ├── ...
|
||||
├── Presets/ # Preset definition files (JSON)
|
||||
│ ├── _template.json # Template for creating new presets
|
||||
│ ├── Poliigon.json # Example preset for Poliigon assets
|
||||
│ └── ... # Other presets
|
||||
├── Project Notes/ # Directory for issue and feature tracking (Markdown files)
|
||||
│ ├── ... # Various planning and note files
|
||||
└── Testfiles/ # Directory containing example input assets for testing
|
||||
└── ... # Example asset ZIPs
|
||||
├── ProjectNotes/ # Directory for developer notes, plans, etc. (Markdown files)
|
||||
│ ├── ...
|
||||
├── PythonCheatsheats/ # Utility Python reference files
|
||||
│ ├── ...
|
||||
├── Testfiles/ # Directory containing example input assets for testing
|
||||
│ ├── ...
|
||||
├── Tickets/ # Directory for issue and feature tracking (Markdown files)
|
||||
│ ├── ...
|
||||
└── utils/ # Utility modules shared across the application
|
||||
├── prediction_utils.py # Utilities for prediction (e.g., used by monitor)
|
||||
└── workspace_utils.py # Utilities for managing processing workspaces
|
||||
```
|
||||
|
||||
**Key Files and Directories:**
|
||||
|
||||
* `asset_processor.py`: Contains the older `AssetProcessor` class. It is kept for reference but is no longer used in the main processing flow orchestrated by `main.py` or the GUI.
|
||||
* `config.py`: Stores global default settings, constants, core rules, and centralized definitions for allowed asset and file types (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`) used for validation, GUI dropdowns, and coloring.
|
||||
* `config/`: Directory containing configuration files, such as `suppliers.json`.
|
||||
* `config/suppliers.json`: A JSON file storing a persistent list of known supplier names, used by the GUI's `SupplierSearchDelegate` for auto-completion.
|
||||
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config.py` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency. An instance of this class is passed to the `ProcessingEngine`.
|
||||
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses. These structures represent the hierarchical processing rules and are the primary data contract passed from the GUI/prediction layer to the processing engine.
|
||||
* `processing_engine.py`: Defines the new `ProcessingEngine` class. This is the core component that executes the processing pipeline for a single asset based *solely* on a provided `SourceRule` object and the static `Configuration`. It contains no internal prediction or fallback logic.
|
||||
* `main.py`: Entry point for the Command-Line Interface (CLI). It handles argument parsing, logging setup, parallel processing orchestration (using `concurrent.futures.ProcessPoolExecutor`), and triggering Blender scripts. It now orchestrates processing by generating or receiving `SourceRule` objects and passing them to the `ProcessingEngine`.
|
||||
* `monitor.py`: Implements the automated directory monitoring feature using the `watchdog` library. Contains the `ZipHandler` class to detect new ZIP files and trigger processing via `main.run_processing`.
|
||||
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6. The GUI is responsible for managing user input, generating and editing the `SourceRule` hierarchy, and interacting with background handlers.
|
||||
* `main_window.py`: Defines the `MainWindow` class, the main application window structure, UI layout, event handling, and menu setup. Integrates the Unified Hierarchical View. Manages GUI-specific logging (`QtLogHandler`).
|
||||
* `unified_view_model.py`: Implements the `QAbstractItemModel` for the Unified Hierarchical View (`QTreeView`). It holds the `SourceRule` hierarchy and provides data and flags for display and inline editing.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations (e.g., for `QComboBox`, `QLineEdit`) used by the Unified View to provide inline editors for rule attributes.
|
||||
* `processing_handler.py`: Defines the `ProcessingHandler` class (runs on a `QThread`). Manages the execution of the `ProcessingEngine` in background processes and communicates status/results back to the GUI.
|
||||
* `prediction_handler.py`: Defines the `PredictionHandler` class (runs on a `QThread`). Generates the initial `SourceRule` hierarchy with predicted values based on input files and the selected preset. Emits a signal with the generated `SourceRule` list for the GUI.
|
||||
* `config/`: Directory containing configuration files.
|
||||
* `app_settings.json`: Stores global default settings, constants, core rules, and centralized definitions for allowed asset and file types (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`) used for validation, GUI elements, and coloring. Replaces the old `config.py`.
|
||||
* `suppliers.json`: A JSON file storing a persistent list of known supplier names, used by the GUI for auto-completion.
|
||||
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config/app_settings.json` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency. An instance of this class is passed to the `ProcessingEngine`.
|
||||
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses. These structures represent the hierarchical processing rules and are the primary data contract passed from the rule generation layer (GUI, Monitor) to the processing engine.
|
||||
* `processing_engine.py`: Defines the `ProcessingEngine` class. This is the core component that executes the processing pipeline for a single asset based *solely* on a provided `SourceRule` object and the static `Configuration`. It contains no internal prediction or fallback logic.
|
||||
* `main.py`: Main entry point for the application. Primarily responsible for initializing and launching the GUI (`gui.main_window.MainWindow`). Contains non-functional/commented-out CLI logic (`run_cli`).
|
||||
* `monitor.py`: Implements the automated directory monitoring feature using `watchdog`. It now processes detected archives asynchronously using a `ThreadPoolExecutor`. It utilizes `utils.prediction_utils.generate_source_rule_from_archive` for rule-based prediction and `utils.workspace_utils.prepare_processing_workspace` for workspace setup before invoking the `ProcessingEngine`.
|
||||
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6. The `MainWindow` acts as a coordinator, delegating functionality to specialized widgets and handlers.
|
||||
* `main_window.py`: Defines the `MainWindow` class. Acts as the main application window and coordinator, connecting signals and slots between different GUI components.
|
||||
* `main_panel_widget.py`: Defines `MainPanelWidget`, containing the primary user controls (source loading, preset selection, rule view/editor integration, processing buttons).
|
||||
* `preset_editor_widget.py`: Defines `PresetEditorWidget` for managing presets (loading, saving, editing).
|
||||
* `log_console_widget.py`: Defines `LogConsoleWidget` for displaying application logs within the GUI.
|
||||
* `rule_editor_widget.py`: Defines `RuleEditorWidget`, which houses the `QTreeView` for displaying the rule hierarchy.
|
||||
* `unified_view_model.py`: Defines `UnifiedViewModel` (`QAbstractItemModel`) for the rule hierarchy view. Holds `SourceRule` data, manages display logic (coloring), handles inline editing requests, and caches configuration data for performance.
|
||||
* `rule_hierarchy_model.py`: Defines `RuleHierarchyModel`, a simpler internal model used by `UnifiedViewModel` to manage the underlying `SourceRule` data structure.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations used by the `UnifiedViewModel` to provide appropriate inline editors (e.g., dropdowns, text boxes) for different rule attributes.
|
||||
* `asset_restructure_handler.py`: Defines `AssetRestructureHandler`. Handles the complex logic of modifying the `SourceRule` hierarchy when a user changes a file's target asset via the GUI, ensuring data integrity. Triggered by signals from the model.
|
||||
* `base_prediction_handler.py`: Defines the abstract `BasePredictionHandler` class, providing a common interface and threading (`QRunnable`) for prediction tasks.
|
||||
* `prediction_handler.py`: Defines `RuleBasedPredictionHandler` (inherits from `BasePredictionHandler`). Generates the initial `SourceRule` hierarchy with predicted values based on input files and the selected preset rules. Runs in a background thread.
|
||||
* `llm_prediction_handler.py`: Defines `LLMPredictionHandler` (inherits from `BasePredictionHandler`). Experimental handler using an LLM for prediction. Runs in a background thread.
|
||||
* `llm_interaction_handler.py`: Defines `LLMInteractionHandler`. Manages the communication details (API calls, etc.) with the LLM service, used by `LLMPredictionHandler`.
|
||||
* `utils/`: Directory containing shared utility modules.
|
||||
* `workspace_utils.py`: Provides functions for managing processing workspaces, such as creating temporary directories and extracting archives (`prepare_processing_workspace`). Used by `main.py` (ProcessingTask) and `monitor.py`.
|
||||
* `prediction_utils.py`: Provides utility functions related to prediction, such as generating a `SourceRule` from an archive (`generate_source_rule_from_archive`), used by `monitor.py`.
|
||||
* `blenderscripts/`: Contains Python scripts (`create_nodegroups.py`, `create_materials.py`) designed to be executed *within* Blender for post-processing.
|
||||
* `Presets/`: Contains supplier-specific configuration files in JSON format, used by the `PredictionHandler` for initial rule generation.
|
||||
* `Presets/`: Contains supplier-specific configuration files in JSON format, used by the `RuleBasedPredictionHandler` for initial rule generation.
|
||||
* `Testfiles/`: Contains example input assets for testing purposes.
|
||||
* `Tickets/`: Directory for issue and feature tracking using Markdown files.
|
||||
* `Tickets/`: Directory for issue and feature tracking using Markdown files.
|
||||
* `Deprecated/`: Contains older code, documentation, and proof-of-concept scripts that are no longer actively used.
|
||||
@@ -6,7 +6,7 @@ This document describes the major classes and modules that form the core of the
|
||||
|
||||
The `ProcessingEngine` class is the new core component responsible for executing the asset processing pipeline for a *single* input asset. Unlike the older `AssetProcessor`, this engine operates *solely* based on a complete `SourceRule` object provided to its `process()` method and the static `Configuration` object passed during initialization. It contains no internal prediction, classification, or fallback logic. Its key responsibilities include:
|
||||
|
||||
* Setting up and cleaning up a temporary workspace for processing.
|
||||
* Setting up and cleaning up a temporary workspace for processing (potentially using `utils.workspace_utils`).
|
||||
* Extracting or copying input files to the workspace.
|
||||
* Processing files based on the explicit rules and predicted values contained within the input `SourceRule`.
|
||||
* Processing texture maps (resizing, format/bit depth conversion, inversion, stats calculation) using parameters from the `SourceRule` or static `Configuration`.
|
||||
@@ -14,10 +14,6 @@ The `ProcessingEngine` class is the new core component responsible for executing
|
||||
* Generating the `metadata.json` file containing details about the processed asset, incorporating information from the `SourceRule`.
|
||||
* Organizing the final output files into the structured library directory.
|
||||
|
||||
## `AssetProcessor` (`asset_processor.py`)
|
||||
|
||||
The `AssetProcessor` class is the older processing engine. It is kept in the codebase for reference but is **no longer used** in the main processing flow orchestrated by `main.py` or the GUI. Its original role was similar to the new `ProcessingEngine`, but it included internal prediction, classification, and fallback logic based on hierarchical rules and static configuration.
|
||||
|
||||
## `Rule Structure` (`rule_structure.py`)
|
||||
|
||||
This module defines the data structures used to represent the hierarchical processing rules:
|
||||
@@ -26,13 +22,13 @@ This module defines the data structures used to represent the hierarchical proce
|
||||
* `AssetRule`: A dataclass representing rules applied at the asset level. It contains nested `FileRule` objects.
|
||||
* `FileRule`: A dataclass representing rules applied at the file level.
|
||||
|
||||
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `asset_type_override`, `item_type`, `item_type_override`, `target_asset_name_override`). Attributes like `asset_type` and `item_type_override` now use string types, which are validated against centralized lists in `config.py`. These structures support serialization (Pickle, JSON) to allow them to be passed between different parts of the application, including across process boundaries.
|
||||
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `asset_type_override`, `item_type`, `item_type_override`, `target_asset_name_override`). Attributes like `asset_type` and `item_type_override` now use string types, which are validated against centralized lists in `config/app_settings.json`. These structures support serialization (Pickle, JSON) to allow them to be passed between different parts of the application, including across process boundaries.
|
||||
|
||||
## `Configuration` (`configuration.py`)
|
||||
|
||||
The `Configuration` class manages the tool's settings. It is responsible for:
|
||||
|
||||
* Loading the core default settings defined in `config.py`.
|
||||
* Loading the core default settings defined in `config/app_settings.json`.
|
||||
* Loading the supplier-specific rules from a selected preset JSON file (`Presets/*.json`).
|
||||
* Merging the core settings and preset rules into a single, unified configuration object.
|
||||
* Validating the loaded configuration to ensure required settings are present.
|
||||
@@ -40,86 +36,147 @@ The `Configuration` class manages the tool's settings. It is responsible for:
|
||||
|
||||
An instance of the `Configuration` class is typically created once per application run (or per processing batch) and passed to the `ProcessingEngine`.
|
||||
|
||||
## `MainWindow` (`gui/main_window.py`)
|
||||
## GUI Components (`gui/`)
|
||||
|
||||
The `MainWindow` class is the main application window for the Graphical User Interface (GUI). It handles the overall UI layout and user interaction:
|
||||
The GUI has been refactored into several key components:
|
||||
|
||||
* Defines the main application window structure and layout using PySide6 widgets.
|
||||
* Arranges the Preset Editor panel (left) and the Unified Hierarchical View (right).
|
||||
* Setting up the menu bar, including the "View" menu for toggling the Log Console.
|
||||
* Connecting user interactions (button clicks, drag-and-drop events, edits in the Unified View) to corresponding methods (slots) within the `MainWindow` or other handler classes.
|
||||
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
|
||||
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
|
||||
* Receiving the initial `SourceRule` hierarchy from the `PredictionHandler` and populating the `UnifiedViewModel`.
|
||||
* Sending the final, potentially user-modified, `SourceRule` list to `main.py` to initiate processing via the `ProcessingEngine`.
|
||||
### `MainWindow` (`gui/main_window.py`)
|
||||
|
||||
## `Unified View Model` (`gui/unified_view_model.py`)
|
||||
The `MainWindow` class acts as the main application window and **coordinator** for the GUI. Its primary responsibilities now include:
|
||||
|
||||
* Setting up the main window structure and menu bar.
|
||||
* Instantiating and arranging the major GUI widgets:
|
||||
* `MainPanelWidget` (containing core controls and the rule editor)
|
||||
* `PresetEditorWidget`
|
||||
* `LogConsoleWidget`
|
||||
* Connecting signals and slots between these widgets, the underlying models (`UnifiedViewModel`), and background handlers (`RuleBasedPredictionHandler`, `LLMPredictionHandler`, `LLMInteractionHandler`).
|
||||
* Managing the overall application state related to GUI interactions (e.g., enabling/disabling controls).
|
||||
* Handling top-level actions like loading sources (drag-and-drop), initiating predictions, and starting the processing task (via `main.ProcessingTask`).
|
||||
* Managing the `QThreadPool` for running background tasks (prediction).
|
||||
* Implementing slots like `_handle_prediction_completion` to update the model/view when prediction results are ready.
|
||||
|
||||
### `MainPanelWidget` (`gui/main_panel_widget.py`)
|
||||
|
||||
This widget contains the central part of the GUI, including:
|
||||
|
||||
* Controls for loading source files/directories.
|
||||
* The preset selection dropdown.
|
||||
* Buttons for initiating prediction and processing.
|
||||
* The `RuleEditorWidget` which houses the hierarchical rule view.
|
||||
|
||||
### `PresetEditorWidget` (`gui/preset_editor_widget.py`)
|
||||
|
||||
This widget provides the interface for managing presets:
|
||||
|
||||
* Loading, saving, and editing preset files (`Presets/*.json`).
|
||||
* Displaying preset rules and settings.
|
||||
|
||||
### `LogConsoleWidget` (`gui/log_console_widget.py`)
|
||||
|
||||
This widget displays application logs within the GUI:
|
||||
|
||||
* Provides a text area for log messages.
|
||||
* Integrates with Python's `logging` system via a custom `QtLogHandler`.
|
||||
* Can be shown/hidden via the main window's "View" menu.
|
||||
|
||||
### `UnifiedViewModel` (`gui/unified_view_model.py`)
|
||||
|
||||
The `UnifiedViewModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
|
||||
|
||||
* Wrap a list of `SourceRule` objects and expose their hierarchical structure (Source -> Asset -> File) to a `QTreeView` (the Unified Hierarchical View).
|
||||
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`, `flags`, `setData`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy and support inline editing of specific attributes (e.g., `supplier_override`, `asset_type_override`, `item_type_override`, `target_asset_name_override`).
|
||||
* Handle the direct restructuring of the underlying `SourceRule` hierarchy when `target_asset_name_override` is edited, including moving `FileRule`s and managing `AssetRule` creation/deletion.
|
||||
* Determine row background colors based on the `asset_type` and `item_type`/`item_type_override` using color metadata from `config.py`.
|
||||
* Hold the `SourceRule` data that is the single source of truth for the GUI's processing rules.
|
||||
* Handle requests for data editing (`setData`) by validating input and updating the underlying `RuleHierarchyModel`. **Note:** Complex restructuring logic (e.g., moving files between assets when `target_asset_name_override` changes) is now delegated to the `AssetRestructureHandler`.
|
||||
* Determine row background colors based on the `asset_type` and `item_type`/`item_type_override` using color metadata from the `Configuration`.
|
||||
* Hold the `SourceRule` data (via `RuleHierarchyModel`) that is the single source of truth for the GUI's processing rules.
|
||||
* Cache configuration data (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`, color maps) during initialization for improved performance in the `data()` method.
|
||||
* Includes the `update_rules_for_sources` method, which intelligently merges new prediction results into the existing model data, preserving user overrides where possible.
|
||||
|
||||
## `Delegates` (`gui/delegates.py`)
|
||||
### `RuleHierarchyModel` (`gui/rule_hierarchy_model.py`)
|
||||
|
||||
A simpler, non-Qt model used internally by `UnifiedViewModel` to manage the list of `SourceRule` objects and provide methods for accessing and modifying the hierarchy.
|
||||
|
||||
### `AssetRestructureHandler` (`gui/asset_restructure_handler.py`)
|
||||
|
||||
This handler contains the complex logic required to modify the `SourceRule` hierarchy when a file's target asset is changed via the GUI's `UnifiedViewModel`. It:
|
||||
|
||||
* Is triggered by a signal (`targetAssetOverrideChanged`) from the `UnifiedViewModel`.
|
||||
* Uses dedicated methods on the `RuleHierarchyModel` (`moveFileRule`, `createAssetRule`, `removeAssetRule`) to safely move `FileRule` objects between `AssetRule`s, creating or removing `AssetRule`s as needed.
|
||||
* Ensures data consistency during these potentially complex restructuring operations.
|
||||
|
||||
### `Delegates` (`gui/delegates.py`)
|
||||
|
||||
This module contains custom `QStyledItemDelegate` implementations used by the Unified Hierarchical View (`QTreeView`) to provide inline editors for specific data types or rule attributes. Examples include delegates for:
|
||||
|
||||
* `ComboBoxDelegate`: For selecting from predefined lists of allowed asset and file types, sourced from `config.py`.
|
||||
* `ComboBoxDelegate`: For selecting from predefined lists of allowed asset and file types, sourced from the `Configuration` (originally from `config/app_settings.json`).
|
||||
* `LineEditDelegate`: For free-form text editing, such as the `target_asset_name_override`.
|
||||
* `SupplierSearchDelegate`: A new delegate for the "Supplier" column. It provides a `QLineEdit` with auto-completion suggestions loaded from `config/suppliers.json` and handles adding/saving new suppliers.
|
||||
* `SupplierSearchDelegate`: For the "Supplier" column. Provides a `QLineEdit` with auto-completion suggestions loaded from `config/suppliers.json` and handles adding/saving new suppliers.
|
||||
|
||||
These delegates handle the presentation and editing of data within the tree view cells, interacting with the `UnifiedViewModel` to get and set data.
|
||||
|
||||
## `ProcessingHandler` (`gui/processing_handler.py`)
|
||||
## Prediction Handlers (`gui/`)
|
||||
|
||||
The `ProcessingHandler` class is designed to run in a separate `QThread` within the GUI. Its purpose is to manage the execution of the main asset processing pipeline using the **`ProcessingEngine`** in the background, preventing the GUI from freezing. It:
|
||||
Prediction logic is handled by classes inheriting from a common base class, running in background threads.
|
||||
|
||||
* Manages a `concurrent.futures.ProcessPoolExecutor` to run individual asset processing tasks (`ProcessingEngine.process()`) in separate worker processes.
|
||||
* Submits processing tasks to the pool, passing the relevant `SourceRule` object and `Configuration` instance to the `ProcessingEngine`.
|
||||
* Monitors task completion and communicates progress, status updates, and results back to the `MainWindow` using Qt signals.
|
||||
* Handles the execution of optional Blender scripts via subprocess calls after asset processing is complete.
|
||||
* Provides logic for cancelling ongoing processing tasks.
|
||||
### `BasePredictionHandler` (`gui/base_prediction_handler.py`)
|
||||
|
||||
## `PredictionHandler` (`gui/prediction_handler.py`)
|
||||
An abstract base class (`QRunnable`) for prediction handlers. It defines the common structure and signals (`prediction_signal`) used by specific predictor implementations. It's designed to be run in a `QThreadPool`.
|
||||
|
||||
The `PredictionHandler` class runs in a separate `QThread` in the GUI and is responsible for generating the initial `SourceRule` hierarchy with predicted values based on the input files and the selected preset *when the rule-based prediction method is selected*. It:
|
||||
### `RuleBasedPredictionHandler` (`gui/prediction_handler.py`)
|
||||
|
||||
* Takes an input source identifier (path), a list of files within that source, and the selected preset name as input.
|
||||
* Uses logic (including accessing preset rules and the `Configuration`'s allowed types) to analyze files and predict initial values for overridable fields in the `SourceRule`, `AssetRule`, and `FileRule` objects (e.g., `supplier_identifier`, `asset_type`, `item_type`, `target_asset_name_override`).
|
||||
* Constructs a `SourceRule` hierarchy for the single input source.
|
||||
* Emits a signal (`rule_hierarchy_ready`) with the input source identifier and the generated `SourceRule` object (within a list) to the `MainWindow` for accumulation and eventual population of the `UnifiedViewModel`.
|
||||
This class (inheriting from `BasePredictionHandler`) is responsible for generating the initial `SourceRule` hierarchy using predefined rules from presets. It:
|
||||
|
||||
## `LLMPredictionHandler` (`gui/llm_prediction_handler.py`)
|
||||
* Takes an input source identifier, file list, and `Configuration` object.
|
||||
* Analyzes files based on regex patterns and rules defined in the loaded preset.
|
||||
* Constructs a `SourceRule` hierarchy with predicted values.
|
||||
* Emits the `prediction_signal` with the generated `SourceRule` object.
|
||||
|
||||
The `LLMPredictionHandler` class is an experimental component that runs in a separate `QThread` and provides an alternative to the `PredictionHandler` by using a Large Language Model (LLM) for prediction. Its key responsibilities include:
|
||||
* Communicating with an external LLM API endpoint (configured via `app_settings.json`).
|
||||
* Sending relevant file information and context to the LLM based on the `llm_predictor_prompt` and `llm_predictor_examples` settings.
|
||||
* Parsing the LLM's response to extract predicted values for `SourceRule`, `AssetRule`, and `FileRule` objects.
|
||||
* Constructs a `SourceRule` hierarchy based on the LLM's interpretation.
|
||||
* Emits a signal (`llm_prediction_ready`) with the input source identifier and the generated `SourceRule` object (within a list) to the `MainWindow` for accumulation and population of the `UnifiedViewModel`.
|
||||
### `LLMPredictionHandler` (`gui/llm_prediction_handler.py`)
|
||||
|
||||
## `UnifiedViewModel` (`gui/unified_view_model.py`)
|
||||
An experimental predictor (inheriting from `BasePredictionHandler`) that uses a Large Language Model (LLM). It:
|
||||
|
||||
*(Note: This section is being moved here from the GUI Internals document for better organization as it's a key component.)*
|
||||
* Takes an input source identifier, file list, and `Configuration` object.
|
||||
* Interacts with the `LLMInteractionHandler` to send data to the LLM and receive predictions.
|
||||
* Parses the LLM response to construct a `SourceRule` hierarchy.
|
||||
* Emits the `prediction_signal` with the generated `SourceRule` object.
|
||||
|
||||
The `UnifiedViewModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
|
||||
* Wrap a list of `SourceRule` objects and expose their hierarchical structure (Source -> Asset -> File) to a `QTreeView` (the Unified Hierarchical View).
|
||||
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`, `flags`, `setData`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy and support inline editing of specific attributes (e.g., `supplier_override`, `asset_type_override`, `item_type_override`, `target_asset_name_override`).
|
||||
* Handle the direct restructuring of the underlying `SourceRule` hierarchy when `target_asset_name_override` is edited, including moving `FileRule`s and managing `AssetRule` creation/deletion.
|
||||
* Determine row background colors based on the `asset_type` and `item_type`/`item_type_override` using color metadata from the `Configuration`.
|
||||
* Hold the `SourceRule` data that is the single source of truth for the GUI's processing rules.
|
||||
* Includes the `update_rules_for_sources` method, which is called by `MainWindow` to update the model's internal `SourceRule` data with new prediction results (from either the `PredictionHandler` or `LLMPredictionHandler`) and trigger the view to refresh.
|
||||
### `LLMInteractionHandler` (`gui/llm_interaction_handler.py`)
|
||||
|
||||
## `ZipHandler` (`monitor.py`)
|
||||
This class manages the specifics of communicating with the configured LLM API:
|
||||
|
||||
The `ZipHandler` is a custom event handler used by the `monitor.py` script, built upon the `watchdog` library. It is responsible for:
|
||||
* Handles constructing prompts based on templates and input data.
|
||||
* Sends requests to the LLM endpoint.
|
||||
* Receives and potentially pre-processes the LLM's response before returning it to the `LLMPredictionHandler`.
|
||||
|
||||
* Detecting file system events, specifically the creation of new `.zip` files, in the monitored input directory.
|
||||
* Validating the filename format of detected ZIPs to extract the intended preset name.
|
||||
* Triggering the main asset processing logic (`main.run_processing`) for valid new ZIP files.
|
||||
* Managing the movement of processed source ZIP files to 'processed' or 'error' directories.
|
||||
## Utility Modules (`utils/`)
|
||||
|
||||
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness. The `SourceRule` object serves as a clear data contract between the GUI/prediction layer and the processing engine.
|
||||
Common utility functions have been extracted into separate modules:
|
||||
|
||||
### `workspace_utils.py`
|
||||
|
||||
Contains functions related to managing the processing workspace:
|
||||
|
||||
* `prepare_processing_workspace`: Creates temporary directories, extracts archive files (ZIP, RAR, 7z), and returns the path to the prepared workspace. Used by `main.ProcessingTask` and `monitor.py`.
|
||||
|
||||
### `prediction_utils.py`
|
||||
|
||||
Contains utility functions supporting prediction tasks:
|
||||
|
||||
* `generate_source_rule_from_archive`: A helper function used by `monitor.py` to perform rule-based prediction directly on an archive file without needing the full GUI setup. It extracts files temporarily, runs prediction logic similar to `RuleBasedPredictionHandler`, and returns a `SourceRule`.
|
||||
|
||||
## Monitor (`monitor.py`)
|
||||
|
||||
The `monitor.py` script implements the directory monitoring feature. It has been refactored to:
|
||||
|
||||
* Use `watchdog` to detect new archive files in the input directory.
|
||||
* Use a `ThreadPoolExecutor` to process detected archives asynchronously in a `_process_archive_task` function.
|
||||
* Within the task, it:
|
||||
* Loads the necessary `Configuration`.
|
||||
* Calls `utils.prediction_utils.generate_source_rule_from_archive` to get the `SourceRule`.
|
||||
* Calls `utils.workspace_utils.prepare_processing_workspace` to set up the workspace.
|
||||
* Instantiates and runs the `ProcessingEngine`.
|
||||
* Handles moving the source archive to 'processed' or 'error' directories.
|
||||
* Cleans up the workspace.
|
||||
|
||||
## Summary
|
||||
|
||||
These key components, along with the refactored GUI structure and new utility modules, work together to provide the tool's functionality. The architecture emphasizes separation of concerns (configuration, rule generation, processing, UI), utilizes background processing for responsiveness (GUI prediction, Monitor tasks), and relies on the `SourceRule` object as the central data structure passed between different stages of the workflow.
|
||||
@@ -6,70 +6,61 @@ The `ProcessingEngine.process()` method orchestrates the following pipeline base
|
||||
|
||||
The pipeline steps are:
|
||||
|
||||
1. **Workspace Setup (`_setup_workspace`)**:
|
||||
* Creates a temporary directory using `tempfile.mkdtemp()` to isolate the processing of the current asset.
|
||||
1. **Workspace Preparation (External)**:
|
||||
* Before the `ProcessingEngine` is invoked, the calling code (e.g., `main.ProcessingTask`, `monitor._process_archive_task`) is responsible for setting up a temporary workspace.
|
||||
* This typically involves using `utils.workspace_utils.prepare_processing_workspace`, which creates a temporary directory and extracts the input source (archive or folder) into it.
|
||||
* The path to this prepared workspace is passed to the `ProcessingEngine` during initialization.
|
||||
|
||||
2. **Input Extraction (`_extract_input`)**:
|
||||
* If the input is a supported archive type (.zip, .rar, .7z), it's extracted into the temporary workspace using the appropriate library (`zipfile`, `rarfile`, or `py7zr`).
|
||||
* If the input is a directory, its contents are copied into the temporary workspace.
|
||||
* Includes basic error handling for invalid or password-protected archives.
|
||||
|
||||
3. **Prediction and Rule Generation (Handled Externally)**:
|
||||
* Before the `ProcessingEngine` is invoked, either the `PredictionHandler` (rule-based) or the `LLMPredictionHandler` (LLM-based) is used (typically triggered by the GUI) to analyze the input files and generate a `SourceRule` object.
|
||||
* This `SourceRule` object contains the predicted classifications (`item_type`, `asset_type`, etc.) and any initial overrides based on the chosen prediction method (preset rules or LLM interpretation).
|
||||
* The GUI allows the user to review and modify these predicted rules before processing begins.
|
||||
* The final, potentially user-modified, `SourceRule` object is the primary input to the `ProcessingEngine`.
|
||||
|
||||
4. **File Inventory (`_inventory_and_classify_files`)**:
|
||||
* Scans the contents of the temporary workspace.
|
||||
* This step primarily inventories the files present. The *classification* itself (determining `item_type`, etc.) has already been performed by the external prediction handler and is stored within the input `SourceRule`. The engine uses the classifications provided in the `SourceRule`.
|
||||
* Stores the file paths and their associated rules from the `SourceRule` in `self.classified_files`.
|
||||
|
||||
5. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
|
||||
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` object and the static configuration from the `Configuration` object. Overrides (like `supplier_identifier`, `asset_type`, and `asset_name_override`), including supplier overrides from the GUI, are taken directly from the `SourceRule`.
|
||||
2. **Prediction and Rule Generation (External)**:
|
||||
* Also handled before the `ProcessingEngine` is invoked.
|
||||
* Either the `RuleBasedPredictionHandler`, `LLMPredictionHandler` (triggered by the GUI), or `utils.prediction_utils.generate_source_rule_from_archive` (used by the Monitor) analyzes the input files and generates a `SourceRule` object.
|
||||
* This `SourceRule` contains predicted classifications and initial overrides.
|
||||
* If using the GUI, the user can modify these rules.
|
||||
* The final `SourceRule` object is the primary input to the `ProcessingEngine.process()` method.
|
||||
|
||||
3. **File Inventory (`_inventory_and_classify_files`)**:
|
||||
* Scans the contents of the *already prepared* temporary workspace.
|
||||
* This step primarily inventories the files present. The *classification* (determining `item_type`, etc.) is taken directly from the input `SourceRule`.
|
||||
* Stores the file paths and their associated rules from the `SourceRule` in `self.classified_files`.
|
||||
|
||||
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
|
||||
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` and the static `Configuration`. Overrides (like `supplier_identifier`, `asset_type`, `asset_name_override`) are taken directly from the `SourceRule`.
|
||||
|
||||
5. **Skip Check**:
|
||||
* If the `overwrite` flag (passed during initialization) is `False`, the tool checks if the final output directory for the determined asset name already exists and contains a `metadata.json` file.
|
||||
* If both exist, processing for this specific asset is skipped, marked as "skipped", and the pipeline moves to the next asset (if processing multiple assets from one source) or finishes.
|
||||
* If the `overwrite` flag is `False`, checks if the final output directory already exists and contains `metadata.json`.
|
||||
* If so, processing for this asset is skipped.
|
||||
|
||||
6. **Map Processing (`_process_maps`)**:
|
||||
* Iterates through the files classified as texture maps for the current asset based on the `SourceRule`. Configuration values used in this step, such as target resolutions, bit depth rules, and output format rules, are retrieved directly from the static `Configuration` object or explicit overrides in the `SourceRule`.
|
||||
* Loads the image using `cv2.imread` (handling grayscale and unchanged flags). Converts BGR to RGB internally for consistency (except for saving non-EXR formats).
|
||||
* Handles Glossiness-to-Roughness inversion if necessary (loads gloss, inverts `1.0 - img/norm`, prioritizes gloss source if both exist).
|
||||
* Resizes the image to target resolutions defined in `IMAGE_RESOULTIONS` (from `Configuration`) using `cv2.resize` (`INTER_LANCZOS4` for downscaling). Upscaling is generally avoided by checks.
|
||||
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (from `Configuration`) or overrides in the `SourceRule`.
|
||||
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a hierarchy of rules defined in the `Configuration` or overrides in the `SourceRule`.
|
||||
* Converts the NumPy array data type appropriately before saving (e.g., float to uint8/uint16 with scaling).
|
||||
* Saves the processed map using `cv2.imwrite` (converting RGB back to BGR if saving to non-EXR formats). Includes fallback logic (e.g., attempting PNG if saving 16-bit EXR fails).
|
||||
* Calculates image statistics (Min/Max/Mean) using `_calculate_image_stats` on normalized float64 data for the `CALCULATE_STATS_RESOLUTION` (from `Configuration`).
|
||||
* Determines the aspect ratio change string (e.g., `"EVEN"`, `"X150"`) using `_normalize_aspect_ratio_change`.
|
||||
* Stores details about each processed map (path, resolution, format, stats, etc.) in `processed_maps_details_asset`.
|
||||
* Iterates through files classified as maps in the `SourceRule`.
|
||||
* Loads images (`cv2.imread`).
|
||||
* Handles Glossiness-to-Roughness inversion.
|
||||
* Resizes images based on `Configuration`.
|
||||
* Determines output bit depth and format based on `Configuration` and `SourceRule`.
|
||||
* Converts data types and saves images (`cv2.imwrite`).
|
||||
* Calculates image statistics.
|
||||
* Stores processed map details.
|
||||
|
||||
7. **Map Merging (`_merge_maps_from_source`)**:
|
||||
* Iterates through the `MAP_MERGE_RULES` defined in the `Configuration`.
|
||||
* Identifies the required *source* map files needed as input for each merge rule based on the classified files in the `SourceRule`.
|
||||
* Determines common resolutions available across the required input maps.
|
||||
* Loads the necessary source map channels for each common resolution (using a helper `_load_and_transform_source` which includes caching).
|
||||
* Converts inputs to normalized float32 (0-1).
|
||||
* Injects default channel values (from rule `defaults` in `Configuration` or overrides in `SourceRule`) if an input channel is missing.
|
||||
* Merges channels using `cv2.merge`.
|
||||
* Determines output bit depth and format based on rules in `Configuration` or overrides in `SourceRule`. Handles potential JPG 16-bit conflict by forcing 8-bit.
|
||||
* Saves the merged map using the `_save_image` helper (includes data type/color space conversions and fallback).
|
||||
* Stores details about each merged map in `merged_maps_details_asset`.
|
||||
* Iterates through `MAP_MERGE_RULES` in `Configuration`.
|
||||
* Identifies required source maps based on `SourceRule`.
|
||||
* Loads source channels, handling missing inputs with defaults from `Configuration` or `SourceRule`.
|
||||
* Merges channels (`cv2.merge`).
|
||||
* Determines output format/bit depth and saves the merged map.
|
||||
* Stores merged map details.
|
||||
|
||||
8. **Metadata File Generation (`_generate_metadata_file`)**:
|
||||
* Collects all determined information for the current asset: base metadata, details from `processed_maps_details_asset` and `merged_maps_details_asset`, list of ignored files, source preset used, etc. This information is derived from the input `SourceRule` and the processing results.
|
||||
* Writes this collected data into the `metadata.json` file within the temporary workspace using `json.dump`.
|
||||
* Collects asset metadata, processed/merged map details, ignored files list, etc., primarily from the `SourceRule` and internal processing results.
|
||||
* Writes data to `metadata.json` in the temporary workspace.
|
||||
|
||||
9. **Output Organization (`_organize_output_files`)**:
|
||||
* Creates the final structured output directory: `<output_base_dir>/<supplier_name>/<asset_name>/`. The `supplier_name` used here is derived from the `SourceRule`, ensuring that supplier overrides from the GUI are respected in the output path.
|
||||
* Creates subdirectories `Extra/`, `Unrecognised/`, and `Ignored/` within the asset directory.
|
||||
* Moves the processed maps, merged maps, model files, `metadata.json`, and files classified as Extra, Unrecognised, or Ignored from the temporary workspace into their respective locations in the final output directory structure.
|
||||
* Creates the final structured output directory (`<output_base_dir>/<supplier_name>/<asset_name>/`), using the supplier name from the `SourceRule`.
|
||||
* Moves processed maps, merged maps, models, metadata, and other classified files from the temporary workspace to the final output directory.
|
||||
|
||||
10. **Workspace Cleanup (`_cleanup_workspace`)**:
|
||||
* Removes the temporary workspace directory and its contents using `shutil.rmtree()`. This is called within a `finally` block to ensure cleanup is attempted even if errors occur during processing.
|
||||
10. **Workspace Cleanup (External)**:
|
||||
* After the `ProcessingEngine.process()` method completes (successfully or with errors), the *calling code* is responsible for cleaning up the temporary workspace directory created in Step 1. This is often done in a `finally` block where `utils.workspace_utils.prepare_processing_workspace` was called.
|
||||
|
||||
11. **(Optional) Blender Script Execution**:
|
||||
* If triggered via CLI arguments (`--nodegroup-blend`, `--materials-blend`) or GUI controls, the orchestrator (`main.py` or `gui/processing_handler.py`) executes the corresponding Blender scripts (`blenderscripts/*.py`) using `subprocess.run` after the `ProcessingEngine.process()` call completes successfully for an asset batch. See `Developer Guide: Blender Integration Internals` for more details.
|
||||
11. **(Optional) Blender Script Execution (External)**:
|
||||
* If triggered (e.g., via CLI arguments or GUI controls), the orchestrating code (e.g., `main.ProcessingTask`) executes the corresponding Blender scripts (`blenderscripts/*.py`) using `subprocess.run` *after* the `ProcessingEngine.process()` call completes successfully.
|
||||
* *Note: Centralized logic for this was intended for `utils/blender_utils.py`, but this utility has not yet been implemented.* See `Developer Guide: Blender Integration Internals` for more details.
|
||||
|
||||
This pipeline, executed by the `ProcessingEngine`, provides a clear and explicit processing flow based on the complete rule set provided by the GUI or other interfaces.
|
||||
@@ -8,103 +8,156 @@ The GUI is built using `PySide6`, which provides Python bindings for the Qt fram
|
||||
|
||||
## Main Window (`gui/main_window.py`)
|
||||
|
||||
The `MainWindow` class is the central component of the GUI application. It is responsible for:
|
||||
The `MainWindow` class acts as the central **coordinator** for the GUI application. It is responsible for:
|
||||
|
||||
* Defining the main application window structure and layout using PySide6 widgets.
|
||||
* Arranging the Preset Editor panel (left) and the **Unified Hierarchical View** (right).
|
||||
* Setting up the menu bar, including the "View" menu for toggling the Log Console.
|
||||
* Connecting user interactions (button clicks, drag-and-drop events, edits in the Unified View) to corresponding methods (slots) within the `MainWindow` or other handler classes.
|
||||
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
|
||||
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`, `LLMPredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
|
||||
* Accumulating prediction results from either the `PredictionHandler` (for rule-based presets) or `LLMPredictionHandler` (for LLM interpretation) for multiple input sources before updating the `UnifiedViewModel`.
|
||||
* Receiving the initial `SourceRule` hierarchy from the appropriate prediction handler (`rule_hierarchy_ready` or `llm_prediction_ready` signals) and calling the `UnifiedViewModel`'s `update_rules_for_sources` method to populate the view model.
|
||||
* Sending the final, potentially user-modified, `SourceRule` list to `main.py` to initiate processing via the `ProcessingEngine`.
|
||||
* Handling the selection in the processing preset dropdown (`self.preset_selector`), distinguishing between standard presets and the special `"- LLM Interpretation -"` value.
|
||||
* Initializing and managing the `self.llm_processing_queue` (a `deque`) when LLM interpretation is selected, adding items to be processed by the LLM.
|
||||
* Implementing the `_start_llm_prediction` method to initiate the LLM prediction process for the queued items by calling `_process_next_llm_item`.
|
||||
* Implementing the `_process_next_llm_item` method, which takes the next item from the `llm_processing_queue`, prepares the necessary data, and starts the `LLMPredictionHandler` thread to process that single item.
|
||||
* Connecting signals from the `LLMPredictionHandler` instance:
|
||||
* `llm_prediction_ready` signal to a slot (e.g., `_on_llm_prediction_ready`) that receives the generated `SourceRule`, updates the `UnifiedViewModel` (via `update_rules_for_sources`), and calls `_process_next_llm_item` to continue processing the queue.
|
||||
* `llm_status_update` signal to a slot (e.g., `_on_llm_status_update`) to display LLM processing status messages in the status bar.
|
||||
* `finished` signal to handle thread cleanup.
|
||||
* Setting up the main application window structure and menu bar.
|
||||
* Instantiating and arranging the major GUI widgets:
|
||||
* `MainPanelWidget` (`gui/main_panel_widget.py`): Contains the core controls, preset selection, and the rule editor.
|
||||
* `PresetEditorWidget` (`gui/preset_editor_widget.py`): Handles preset loading, saving, and editing.
|
||||
* `LogConsoleWidget` (`gui/log_console_widget.py`): Displays application logs.
|
||||
* Instantiating key models and handlers:
|
||||
* `UnifiedViewModel` (`gui/unified_view_model.py`): The model for the rule hierarchy view.
|
||||
* `LLMInteractionHandler` (`gui/llm_interaction_handler.py`): Manages communication with the LLM service.
|
||||
* Connecting signals and slots between these components to orchestrate the application flow.
|
||||
* Handling top-level user interactions like drag-and-drop for loading sources (`add_input_paths`). This method now handles the "placeholder" state (no preset selected) by scanning directories or inspecting archives (ZIP) and creating placeholder `SourceRule`/`AssetRule`/`FileRule` objects to immediately populate the `UnifiedViewModel` with the file structure.
|
||||
* Initiating predictions based on the selected preset mode (Rule-Based or LLM) when presets change or sources are added.
|
||||
* Starting the processing task (`_on_process_requested`): This slot now filters the `SourceRule` list obtained from the `UnifiedViewModel`, excluding sources where no asset has a `Target Asset` name assigned, before emitting the `start_backend_processing` signal. It also manages enabling/disabling controls.
|
||||
* Managing the `QThreadPool` for running background prediction tasks (`RuleBasedPredictionHandler`, `LLMPredictionHandler`).
|
||||
* Implementing slots to handle results from background tasks:
|
||||
* `_handle_prediction_completion(source_id, source_rule_list)`: Receives results from either prediction handler via the `prediction_signal`. It calls `self.unified_view_model.update_rules_for_sources()` to update the view model, preserving user overrides where possible. For LLM predictions, it also triggers processing the next item in the queue.
|
||||
* Slots to handle status updates from the LLM handler.
|
||||
|
||||
## Threading and Background Tasks
|
||||
|
||||
To keep the UI responsive during intensive operations like asset processing and rule prediction, the GUI utilizes background threads managed by `QThread`.
|
||||
To keep the UI responsive, prediction tasks run in background threads managed by a `QThreadPool`.
|
||||
|
||||
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline using the **`ProcessingEngine`** for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool, passing the relevant `SourceRule` object and `Configuration` instance to the `ProcessingEngine`. It monitors task completion and communicates progress, status updates, and results back to the `MainWindow` on the main UI thread using Qt signals. It also handles the execution of optional Blender scripts via subprocess calls after processing.
|
||||
* **`PredictionHandler` (`gui/prediction_handler.py`):** Runs in a `QThread` when a rule-based preset is selected. Generates the initial `SourceRule` hierarchy based on preset rules and emits `rule_hierarchy_ready`.
|
||||
* **`LLMPredictionHandler` (`gui/llm_prediction_handler.py`):** Runs in a `QThread` when "- LLM Interpretation -" is selected. Communicates with the LLM API, parses the response, generates the `SourceRule` hierarchy for a *single* input item at a time, and emits `llm_prediction_ready` and `llm_status_update`.
|
||||
* **`BasePredictionHandler` (`gui/base_prediction_handler.py`):** An abstract `QRunnable` base class defining the common interface and signals (`prediction_signal`, `status_signal`) for prediction tasks.
|
||||
* **`RuleBasedPredictionHandler` (`gui/prediction_handler.py`):** Inherits from `BasePredictionHandler`. Runs as a `QRunnable` in the thread pool when a rule-based preset is selected. Generates the `SourceRule` hierarchy based on preset rules and emits `prediction_signal`.
|
||||
* **`LLMPredictionHandler` (`gui/llm_prediction_handler.py`):** Inherits from `BasePredictionHandler`. Runs as a `QRunnable` in the thread pool when "- LLM Interpretation -" is selected. Interacts with `LLMInteractionHandler`, parses the response, generates the `SourceRule` hierarchy for a *single* input item, and emits `prediction_signal` and `status_signal`.
|
||||
* **`LLMInteractionHandler` (`gui/llm_interaction_handler.py`):** Manages the communication with the LLM service. This handler itself may perform network operations but typically runs synchronously within the `LLMPredictionHandler`'s thread.
|
||||
|
||||
*(Note: The actual processing via `ProcessingEngine` is now handled by `main.ProcessingTask`, which runs in a separate process managed outside the GUI's direct threading model, though the GUI initiates it).*
|
||||
|
||||
## Communication (Signals and Slots)
|
||||
|
||||
Communication between the main UI thread (`MainWindow`) and the background threads (`ProcessingHandler`, `PredictionHandler`, `LLMPredictionHandler`) relies heavily on Qt's signals and slots mechanism. This is a thread-safe way for objects in different threads to communicate.
|
||||
Communication between the `MainWindow` (main UI thread) and the background prediction tasks relies on Qt's signals and slots.
|
||||
|
||||
* Background handlers emit signals to indicate events (e.g., progress updated, file status changed, task finished, prediction ready, LLM status update).
|
||||
* The `MainWindow` connects slots (methods) to these signals. When a signal is emitted, the connected slot is invoked on the thread that owns the receiving object (the main UI thread for `MainWindow`), ensuring UI updates happen safely. Key signals/slots related to LLM integration:
|
||||
* `LLMPredictionHandler.llm_prediction_ready(source_id, source_rule_list)` -> `MainWindow._on_llm_prediction_ready(source_id, source_rule_list)` (updates model via `update_rules_for_sources`, processes next queue item)
|
||||
* `LLMPredictionHandler.llm_status_update(message)` -> `MainWindow._on_llm_status_update(message)` (updates status bar)
|
||||
* `LLMPredictionHandler.finished` -> `MainWindow._on_llm_thread_finished` (handles thread cleanup)
|
||||
* Prediction handlers (`RuleBasedPredictionHandler`, `LLMPredictionHandler`) emit signals from the `BasePredictionHandler`:
|
||||
* `prediction_signal(source_id, source_rule_list)`: Indicates prediction for a source is complete.
|
||||
* `status_signal(message)`: Provides status updates (primarily from LLM handler).
|
||||
* The `MainWindow` connects slots to these signals:
|
||||
* `prediction_signal` -> `MainWindow._handle_prediction_completion(source_id, source_rule_list)`
|
||||
* `status_signal` -> `MainWindow._on_status_update(message)` (updates status bar)
|
||||
* Signals from the `UnifiedViewModel` (`dataChanged`, `layoutChanged`) trigger updates in the `QTreeView`.
|
||||
* Signals from the `UnifiedViewModel` (`targetAssetOverrideChanged`) trigger the `AssetRestructureHandler`.
|
||||
|
||||
## Preset Editor
|
||||
## Preset Editor (`gui/preset_editor_widget.py`)
|
||||
|
||||
The GUI includes an integrated preset editor panel. This allows users to interactively create, load, modify, and save preset `.json` files directly within the application. The editor typically uses standard UI widgets to display and edit the key fields of the preset structure.
|
||||
The `PresetEditorWidget` provides a dedicated interface for managing presets. It handles loading, displaying, editing, and saving preset `.json` files. It communicates with the `MainWindow` (e.g., via signals) when a preset is loaded or saved.
|
||||
|
||||
## Unified Hierarchical View (`gui/unified_view_model.py`, `gui/delegates.py`)
|
||||
## Unified Hierarchical View
|
||||
|
||||
## Unified Hierarchical View (`gui/unified_view_model.py`, `gui/delegates.py`, `gui/main_window.py`)
|
||||
The core rule editing interface is built around a `QTreeView` managed within the `MainPanelWidget`, using a custom model and delegates.
|
||||
|
||||
The core of the GUI's rule editing interface is the Unified Hierarchical View, implemented using a `QTreeView` with a custom model and delegates. This view is managed within the `MainWindow`.
|
||||
* **`UnifiedViewModel` (`gui/unified_view_model.py`):** Implements `QAbstractItemModel`.
|
||||
* Wraps the `RuleHierarchyModel` to expose the `SourceRule` list (Source -> Asset -> File) to the `QTreeView`.
|
||||
* Provides data for display and flags for editing.
|
||||
* **Handles `setData` requests:** Validates input and updates the underlying `RuleHierarchyModel`. Crucially, it **delegates** complex restructuring (when `target_asset_name_override` changes) to the `AssetRestructureHandler` by emitting the `targetAssetOverrideChanged` signal.
|
||||
* **Row Coloring:** Provides data for `Qt.ForegroundRole` (text color) based on the `item_type` and the colors defined in `config/app_settings.json`. Provides data for `Qt.BackgroundRole` based on calculating a 30% darker shade of the parent asset's background color.
|
||||
* **Caching:** Caches configuration data (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`, color maps) in `__init__` for performance.
|
||||
* **`update_rules_for_sources` Method:** Intelligently merges new prediction results or placeholder rules into the existing model data, preserving user overrides where applicable.
|
||||
* *(Note: The previous concept of switching between "simple" and "detailed" display modes has been removed. The model always represents the full detailed structure.)*
|
||||
* **`RuleHierarchyModel` (`gui/rule_hierarchy_model.py`):** A non-Qt model holding the actual list of `SourceRule` objects. Provides methods for accessing and modifying the hierarchy (used by `UnifiedViewModel` and `AssetRestructureHandler`).
|
||||
* **`AssetRestructureHandler` (`gui/asset_restructure_handler.py`):** Contains the logic to modify the `RuleHierarchyModel` when a file's target asset is changed. It listens for the `targetAssetOverrideChanged` signal from the `UnifiedViewModel` and uses methods on the `RuleHierarchyModel` (`moveFileRule`, `createAssetRule`, `removeAssetRule`) to perform the restructuring safely.
|
||||
* **`Delegates` (`gui/delegates.py`):** Custom `QStyledItemDelegate` implementations provide inline editors:
|
||||
* **`ComboBoxDelegate`:** For selecting predefined types (from `Configuration`).
|
||||
* **`LineEditDelegate`:** For free-form text editing.
|
||||
* **`SupplierSearchDelegate`:** For supplier names with auto-completion (using `config/suppliers.json`).
|
||||
|
||||
* **`Unified View Model` (`gui/unified_view_model.py`):** This class implements a `QAbstractItemModel` to expose the structure of a list of `SourceRule` objects (Source -> Asset -> File) to the `QTreeView`. It holds the `SourceRule` data that is the single source of truth for the GUI's processing rules. It provides data and flags for display in multiple columns and supports inline editing of specific rule attributes (e.g., asset type, item type override, target asset name override) by interacting with delegates.
|
||||
* **Column Order and Resizing:** The view currently displays the following columns in order: Name, Target Asset, Supplier, Asset Type, Item Type. The "Target Asset" column is set to stretch to fill available space, while other columns resize to their contents. The previous "Status" and "Output Path" columns have been removed.
|
||||
* **Direct Model Restructuring:** The `setData` method now includes logic to directly restructure the underlying `SourceRule` hierarchy when the `target_asset_name_override` field of a `FileRule` is edited. This involves moving the `FileRule` to a different `AssetRule` (creating a new one if necessary) and removing the old `AssetRule` if it becomes empty. This replaces the previous mechanism of re-running prediction after an edit.
|
||||
* **Row Coloring:** Row background colors are dynamically determined based on the `asset_type` (for `AssetRule`s) and `item_type` or `item_type_override` (for `FileRule`s), using the color metadata defined in the `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` dictionaries sourced from the configuration loaded by `configuration.py` (which includes data from `config/app_settings.json`). `SourceRule` rows have a fixed color.
|
||||
* **`Delegates` (`gui/delegates.py`):** This module contains custom `QStyledItemDelegate` implementations used by the `QTreeView` to provide inline editors for specific data types or rule attributes.
|
||||
* **`ComboBoxDelegate`:** Used for selecting from predefined lists (e.g., allowed asset types, allowed file types sourced from the configuration loaded by `configuration.py`).
|
||||
* **`LineEditDelegate`:** Used for free-form text editing (e.g., target asset name override).
|
||||
* **`SupplierSearchDelegate`:** A new delegate used for the "Supplier" column. It provides a `QLineEdit` with auto-completion suggestions loaded from `config/suppliers.json`. It also handles adding new, unique supplier names entered by the user to the list and saving the updated list back to the JSON file.
|
||||
|
||||
The appropriate prediction handler (`PredictionHandler` or `LLMPredictionHandler`) generates the initial `SourceRule` hierarchy (either for all sources at once or one source at a time for LLM). The `MainWindow` receives this via a signal (`rule_hierarchy_ready` or `llm_prediction_ready`) and calls the `UnifiedViewModel`'s `update_rules_for_sources(source_id, source_rule_list)` method. This method updates the model's internal data structure with the new or updated `SourceRule` object(s) for the given `source_id` and emits the necessary signals (`dataChanged`, `layoutChanged`) to refresh the `QTreeView` display. Edits made in the view directly modify the attributes of the underlying rule objects in the `SourceRule` hierarchy held by the model, with the `UnifiedViewModel` handling the necessary model restructuring and signal emission for view updates.
|
||||
|
||||
**Data Flow Diagram (GUI Rule Management):**
|
||||
**Data Flow Diagram (GUI Rule Management - Refactored):**
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[User Input (Drag/Drop, Preset Select)] --> B(MainWindow);
|
||||
B -- Selects Preset/LLM --> B;
|
||||
B -- Starts --> C{Prediction Handler (Rule or LLM)};
|
||||
C -- rule_hierarchy_ready / llm_prediction_ready --> B;
|
||||
B -- Calls update_rules_for_sources(source_id, rules) --> D(UnifiedViewModel);
|
||||
D -- Emits dataChanged/layoutChanged --> E(QTreeView - Unified View);
|
||||
B -- Sets Model --> E;
|
||||
E -- Displays Data from --> D;
|
||||
E -- Uses Delegates from --> F(Delegates);
|
||||
F -- Interact with --> D;
|
||||
User -- Edits Rules via --> E;
|
||||
E -- Updates Data in --> D;
|
||||
B -- Triggers Processing with Final SourceRule List --> G(main.py / ProcessingHandler);
|
||||
graph TD
|
||||
subgraph MainWindow [MainWindow Coordinator]
|
||||
direction LR
|
||||
MW_Input[User Input (Drag/Drop, Preset Select)] --> MW(MainWindow);
|
||||
MW -- Initiates --> PredPool{QThreadPool};
|
||||
MW -- Connects Signals --> VM(UnifiedViewModel);
|
||||
MW -- Connects Signals --> ARH(AssetRestructureHandler);
|
||||
MW -- Owns/Manages --> MPW(MainPanelWidget);
|
||||
MW -- Owns/Manages --> PEW(PresetEditorWidget);
|
||||
MW -- Owns/Manages --> LCW(LogConsoleWidget);
|
||||
MW -- Owns/Manages --> LLMIH(LLMInteractionHandler);
|
||||
end
|
||||
|
||||
subgraph MainPanel [MainPanelWidget]
|
||||
direction TB
|
||||
MPW_UI[UI Controls (Load, Predict, Process Btns)];
|
||||
MPW_UI --> MPW;
|
||||
MPW -- Contains --> REW(RuleEditorWidget);
|
||||
end
|
||||
|
||||
subgraph RuleEditor [RuleEditorWidget]
|
||||
direction TB
|
||||
REW -- Contains --> TV(QTreeView - Rule View);
|
||||
end
|
||||
|
||||
subgraph Prediction [Background Prediction]
|
||||
direction TB
|
||||
PredPool -- Runs --> RBP(RuleBasedPredictionHandler);
|
||||
PredPool -- Runs --> LLMP(LLMPredictionHandler);
|
||||
LLMP -- Uses --> LLMIH;
|
||||
RBP -- prediction_signal --> MW;
|
||||
LLMP -- prediction_signal --> MW;
|
||||
LLMP -- status_signal --> MW;
|
||||
end
|
||||
|
||||
subgraph ModelView [Model/View Components]
|
||||
direction TB
|
||||
TV -- Sets Model --> VM;
|
||||
TV -- Displays Data From --> VM;
|
||||
TV -- Uses Delegates --> Del(Delegates);
|
||||
UserEdit[User Edits Rules] --> TV;
|
||||
TV -- setData --> VM;
|
||||
VM -- Wraps --> RHM(RuleHierarchyModel);
|
||||
VM -- dataChanged/layoutChanged --> TV;
|
||||
VM -- targetAssetOverrideChanged --> ARH;
|
||||
ARH -- Modifies --> RHM;
|
||||
Del -- Get/Set Data --> VM;
|
||||
end
|
||||
|
||||
MW -- _handle_prediction_completion --> VM;
|
||||
MW -- Triggers Processing --> ProcTask(main.ProcessingTask);
|
||||
|
||||
%% Connections between subgraphs
|
||||
MPW --> MW;
|
||||
PEW --> MW;
|
||||
LCW --> MW;
|
||||
VM --> MW;
|
||||
ARH --> MW;
|
||||
LLMIH --> MW;
|
||||
REW --> MPW;
|
||||
```
|
||||
|
||||
## Application Styling
|
||||
|
||||
The application style is explicitly set to 'Fusion' in `gui/main_window.py` to provide a more consistent look and feel across different operating systems. A custom `QPalette` is also applied to the application to adjust default colors within the 'Fusion' style.
|
||||
The application style is explicitly set to 'Fusion' in `gui/main_window.py`. A custom `QPalette` adjusts default colors.
|
||||
|
||||
## Logging
|
||||
## Logging (`gui/log_console_widget.py`)
|
||||
|
||||
A custom `QtLogHandler` is used to redirect log messages from the standard Python `logging` module to a text area or console widget within the GUI, allowing users to see detailed application output and errors.
|
||||
The `LogConsoleWidget` displays logs captured by a custom `QtLogHandler` from Python's `logging` module.
|
||||
|
||||
## Cancellation
|
||||
|
||||
The GUI provides a "Cancel" button to stop ongoing processing. The `ProcessingHandler` implements logic to handle cancellation requests. This typically involves setting an internal flag and attempting to shut down the `ProcessPoolExecutor`. However, it's important to note that this does not immediately terminate worker processes that are already executing; it primarily prevents new tasks from starting and stops processing results from completed futures once the cancellation flag is checked.
|
||||
The GUI provides a "Cancel" button. Cancellation logic for the actual processing is now likely handled within the `main.ProcessingTask` or the code that manages it, as the `ProcessingHandler` has been removed. The GUI button would signal this external task manager.
|
||||
|
||||
## GUI Configuration Editor (`gui/config_editor_dialog.py`)
|
||||
|
||||
A dedicated dialog, implemented in `gui/config_editor_dialog.py`, provides a graphical interface for editing the core application settings stored in `config/app_settings.json`.
|
||||
A dedicated dialog for editing `config/app_settings.json`.
|
||||
|
||||
* **Functionality:** This dialog loads the current content of `config/app_settings.json` and presents it in a tabbed layout (e.g., "General", "Output & Naming") using standard GUI widgets mapped to the JSON structure. It supports editing basic fields, tables for definitions (`FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`), and a list/detail view for merge rules (`MAP_MERGE_RULES`). The definitions tables include dynamic color editing features.
|
||||
* **Limitations:** Currently, editing complex fields like `IMAGE_RESOLUTIONS` or the full details of `MAP_MERGE_RULES` via the UI is not fully supported.
|
||||
* **Integration:** The `MainWindow` is responsible for creating and displaying an instance of this dialog when the user selects the "Edit" -> "Preferences..." menu option.
|
||||
* **Persistence:** Changes saved via this editor are written directly to the `config/app_settings.json` file, ensuring they persist across application sessions. However, the `Configuration` class loads settings at application startup, so a restart is required for changes made in the editor to take effect in the application's processing logic.
|
||||
* **Functionality:** Loads `config/app_settings.json`, presents in tabs, allows editing basic fields, definitions tables (with color editing), and merge rules list/detail.
|
||||
* **Limitations:** Editing complex fields like `IMAGE_RESOLUTIONS` or full `MAP_MERGE_RULES` details might still be limited.
|
||||
* **Integration:** Launched by `MainWindow` ("Edit" -> "Preferences...").
|
||||
* **Persistence:** Saves changes to `config/app_settings.json`. Requires application restart for changes to affect processing logic loaded by the `Configuration` class.
|
||||
|
||||
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness. The Unified Hierarchical View centralizes rule management in the GUI, and the `SourceRule` object serves as a clear data contract passed to the processing engine.
|
||||
The refactored GUI separates concerns into distinct widgets and handlers, coordinated by the `MainWindow`. Background tasks use `QThreadPool` and `QRunnable`. The `UnifiedViewModel` focuses on data presentation and simple edits, delegating complex restructuring to the `AssetRestructureHandler`.
|
||||
@@ -4,30 +4,41 @@ This document provides technical details about the implementation of the Directo
|
||||
|
||||
## Overview
|
||||
|
||||
The `monitor.py` script provides an automated way to process assets by monitoring a specified input directory for new ZIP files. It is built using the `watchdog` library.
|
||||
The `monitor.py` script provides an automated way to process assets by monitoring a specified input directory for new archive files. It has been refactored to use a `ThreadPoolExecutor` for asynchronous processing.
|
||||
|
||||
## Key Components
|
||||
|
||||
* **`watchdog` Library:** The script relies on the `watchdog` library for monitoring file system events. Specifically, it uses a `PollingObserver` to watch the `INPUT_DIR` for changes.
|
||||
* **`ZipHandler` Class:** This is a custom event handler class defined within `monitor.py`. It inherits from a `watchdog` event handler class (likely `FileSystemEventHandler` or similar, though not explicitly stated in the source text, it's the standard pattern). Its primary method of interest is the one that handles file creation events (`on_created`).
|
||||
* **`main.run_processing`:** The monitor script triggers the main asset processing logic by calling the `run_processing` function from the `main.py` module.
|
||||
* **`watchdog` Library:** Used for monitoring file system events (specifically file creation) in the `INPUT_DIR`. A `PollingObserver` is typically used.
|
||||
* **`concurrent.futures.ThreadPoolExecutor`:** Manages a pool of worker threads to process detected archives concurrently. The number of workers can often be configured (e.g., via `NUM_WORKERS` environment variable).
|
||||
* **`_process_archive_task` Function:** The core function executed by the thread pool for each detected archive. It encapsulates the entire processing workflow for a single archive.
|
||||
* **`utils.prediction_utils.generate_source_rule_from_archive`:** A utility function called by `_process_archive_task` to perform rule-based prediction directly on the archive file and generate the necessary `SourceRule` object.
|
||||
* **`utils.workspace_utils.prepare_processing_workspace`:** A utility function called by `_process_archive_task` to create a temporary workspace and extract the archive contents into it.
|
||||
* **`ProcessingEngine` (`processing_engine.py`):** The core engine instantiated and run within `_process_archive_task` to perform the actual asset processing based on the generated `SourceRule`.
|
||||
* **`Configuration` (`configuration.py`):** Loaded within `_process_archive_task` based on the preset derived from the archive filename.
|
||||
|
||||
## Functionality Details
|
||||
## Functionality Details (Asynchronous Workflow)
|
||||
|
||||
1. **Watching:** A `PollingObserver` is set up to monitor the directory specified by the `INPUT_DIR` environment variable. Polling is used, checking for changes at a frequency defined by `POLL_INTERVAL`.
|
||||
2. **Event Handling:** The `ZipHandler` is attached to the observer. When a file is created in the monitored directory, the `on_created` method of the `ZipHandler` is triggered.
|
||||
3. **ZIP File Detection:** The `on_created` method checks if the newly created file is a `.zip` file.
|
||||
4. **Filename Parsing:** If it's a ZIP file, the script expects the filename to follow a specific format: `[preset]_filename.zip`. It uses a regular expression (`PRESET_FILENAME_REGEX`, likely defined in `config.py` or similar) to extract the `[preset]` part from the filename.
|
||||
5. **Preset Validation:** It validates whether the extracted `preset` name corresponds to an existing preset JSON file in the `Presets/` directory.
|
||||
6. **Triggering Processing:** If the preset is valid, the `monitor.py` script calls `main.run_processing`, passing the path to the detected ZIP file and the extracted preset name. This initiates the main asset processing pipeline for that single asset. A `PROCESS_DELAY` can be configured to wait before triggering processing, potentially allowing large files to finish copying.
|
||||
7. **Source ZIP Management:** After the processing initiated by `main.run_processing` completes, the original source `.zip` file is moved to either the `PROCESSED_DIR` (if processing was successful or skipped) or the `ERROR_DIR` (if processing failed or the preset was invalid).
|
||||
1. **Watching:** A `watchdog` observer monitors the `INPUT_DIR` for file creation events.
|
||||
2. **Event Handling:** When a file is created, an event handler (e.g., `on_created` method) is triggered.
|
||||
3. **Archive Detection:** The handler checks if the new file is a supported archive type (e.g., `.zip`, `.rar`, `.7z`).
|
||||
4. **Filename Parsing:** If it's a supported archive, the script attempts to parse the filename to extract the intended preset name (e.g., using a regex like `[preset]_filename.ext`).
|
||||
5. **Preset Validation:** The extracted preset name is validated against existing preset files (`Presets/*.json`).
|
||||
6. **Task Submission:** If the preset is valid, the path to the archive file and the validated preset name are submitted as a task to the `ThreadPoolExecutor`, which will eventually run the `_process_archive_task` function with these arguments in a worker thread.
|
||||
7. **`_process_archive_task` Execution (Worker Thread):**
|
||||
* **Load Configuration:** Loads the `Configuration` object using the provided preset name.
|
||||
* **Generate SourceRule:** Calls `utils.prediction_utils.generate_source_rule_from_archive`, passing the archive path and `Configuration`. This utility handles temporary extraction (if needed internally) and rule-based prediction, returning the `SourceRule`.
|
||||
* **Prepare Workspace:** Calls `utils.workspace_utils.prepare_processing_workspace`, passing the archive path. This creates a unique temporary directory and extracts the archive contents. It returns the path to the prepared workspace. This step should ideally be wrapped in a `try...finally` block to ensure cleanup.
|
||||
* **Instantiate Engine:** Creates an instance of the `ProcessingEngine`, passing the loaded `Configuration` and the prepared workspace path.
|
||||
* **Run Processing:** Calls the `ProcessingEngine.process()` method, passing the generated `SourceRule`.
|
||||
* **Handle Results:** Based on the success or failure of the processing, moves the original source archive file from `INPUT_DIR` to either `PROCESSED_DIR` or `ERROR_DIR`.
|
||||
* **Cleanup Workspace:** Ensures the temporary workspace directory created by `prepare_processing_workspace` is removed (e.g., in the `finally` block).
|
||||
|
||||
## Configuration
|
||||
|
||||
The monitor's behavior is primarily controlled by environment variables, which are read by the `monitor.py` script. These include `INPUT_DIR`, `OUTPUT_DIR`, `PROCESSED_DIR`, `ERROR_DIR`, `LOG_LEVEL`, `POLL_INTERVAL`, and `NUM_WORKERS`.
|
||||
The monitor's behavior is controlled by environment variables or configuration settings, likely including `INPUT_DIR`, `OUTPUT_DIR`, `PROCESSED_DIR`, `ERROR_DIR`, `LOG_LEVEL`, `POLL_INTERVAL`, and potentially `NUM_WORKERS` to control the size of the `ThreadPoolExecutor`.
|
||||
|
||||
## Limitations
|
||||
|
||||
* The current implementation of the directory monitor does *not* support triggering the optional Blender script execution after processing. This post-processing step is only available when running the tool via the CLI or GUI.
|
||||
* The monitor likely still does *not* support triggering optional Blender script execution post-processing, as this integration point was complex and potentially removed or not yet reimplemented in the refactored workflow.
|
||||
|
||||
Understanding the interaction between `watchdog`, the `ZipHandler`, and the call to `main.run_processing` is key to debugging or modifying the directory monitoring functionality.
|
||||
Understanding the asynchronous nature, the role of the `ThreadPoolExecutor`, the `_process_archive_task` function, and the reliance on utility modules (`prediction_utils`, `workspace_utils`) is key to debugging or modifying the directory monitoring functionality.
|
||||
Reference in New Issue
Block a user