# product_context

content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"core_application_entry_and_utilities": {"description": "Serves as the main entry point for launching the application (GUI and a non-functional CLI stub) and provides common, reusable utility functions. Initializes the application, manages concurrent processing tasks for the GUI using QThreadPool, and offers shared functionalities like file/path manipulation (pattern-based path generation, name sanitization, incrementing value calculation), temporary workspace management (archive extraction, directory copying), and file hashing (SHA256).", "key_responsibilities": ["Application initialization and launch (GUI via PySide6, CLI via argparse).", "Orchestration of GUI-initiated concurrent processing tasks (main.App, main.ProcessingTask).", "Provision of shared utility functions for path manipulation (utils/path_utils.py), workspace setup (utils/workspace_utils.py), and hashing (utils/hash_utils.py)."], "key_modules_or_patterns": ["main.py (Application Entry & GUI Task Management - System Pattern ID: 24)", "utils/ (Core Utilities Collection - System Pattern ID: 23)", "utils/path_utils.py", "utils/workspace_utils.py", "utils/hash_utils.py"], "related_system_pattern_ids": [23, 24]}}, "hierarchical_rule_system": {"description": "The Asset Processor Tool utilizes a Hierarchical Rule System to provide dynamic and fine-grained control over asset processing. This system allows for overrides of static preset configurations at multiple levels.", "structure": {"SourceRule": "Top-level rule for an entire input source. Defines supplier info, high-level sorting, input path, preset, and contains AssetRules.", "AssetRule": "Mid-level rule for an individual asset. Defines asset name, type, common metadata, and contains FileRules.", "FileRule": "Lowest-level rule for a specific file within an asset. Defines file path, item type, overrides for target asset name/resolution, channel merge instructions, output format, and contains ProcessingItems."}, "purpose": "Enables customized processing instructions, populated by the Prediction System and consumed by the Processing Engine, allowing for flexible adaptation to diverse asset characteristics.", "key_file": "rule_structure.py"}, "automated_processing_capabilities": {"directory_monitoring": {"description": "The system includes a Directory Monitor System (implemented in monitor.py) that enables automated, unattended processing of assets. It watches a designated input directory for new archives (zip, rar, 7z).", "rule_generation": "Upon detecting a new archive, the monitor automatically generates processing rules, bypassing the need for manual GUI interaction for these assets.", "engine_invocation": "It then invokes the Processing Engine to process the assets according to the auto-generated rules.", "concurrency": "Uses a ThreadPoolExecutor to handle multiple incoming assets concurrently.", "configuration": "Key aspects like monitored directory, output paths, polling interval, and processing delay are configurable via environment variables."}}, "blender_integration_system": {"description": "Extends the asset processing pipeline with functionalities executed within Blender. This includes Python scripts for post-processing tasks like automated node group and material creation from processed assets, and a Blender add-on for merging materials.", "key_components": [{"name": "Blender Node Group Generation", "details": "Script ([`blenderscripts/create_nodegroups.py`](blenderscripts/create_nodegroups.py:1)) that creates `PBRSET` (parent) and `PBRTYPE` (child) node groups in a Blender file based on `metadata.json` from processed assets. Textures are loaded and parameters set automatically."}, {"name": "Blender Material Creation", "details": "Script ([`blenderscripts/create_materials.py`](blenderscripts/create_materials.py:1)) that creates Blender materials, linking them to `PBRSET` node groups expected to be in a separate library `.blend` file. Sets previews and viewport properties."}, {"name": "Material Merger Add-on", "details": "Add-on ([`blender_addon/material_merger/`](blender_addon/material_merger/)) providing a UI to merge two materials by combining their node trees using a `MaterialMerge` utility node group."}], "purpose": "To streamline the process of using Asset Processor Tool outputs directly within Blender, automating the setup of materials and node groups, and providing tools for further material customization."}}


# active_context

content:
{"current_task": "Code review and refactoring plan for GUI file table", "issues_identified": ["File list discrepancies post-LLM processing", "Persistent empty asset rows after renaming/restructuring"], "key_modules_for_review": ["gui/unified_view_model.py", "gui/asset_restructure_handler.py", "gui/llm_prediction_handler.py", "gui/prediction_handler.py", "gui/main_window.py", "gui/main_panel_widget.py", "rule_structure.py"], "proposed_plan_details": "Detailed plan covering code review phases, refactoring strategies for identified issues, and general robustness improvements.", "current_subtask": "Develop Refactoring Strategies (Phase 2)", "subtask_objective": "Develop and propose specific refactoring strategies based on Phase 1 findings to address file list discrepancies, persistent empty rows, and improve robustness/maintainability."}


# decisions

timestamp: 2025-05-14 20:42:53.580194
summary: Adopt a Staged Pipeline Architecture for Asset Processing
tags: ["architecture", "pipeline_design", "processing_engine"]
rationale:
A staged architecture (pre-item, core item processing, post-item) allows for modularity, clear separation of concerns, and easier modification or extension of the processing workflow. The PipelineOrchestrator manages the execution of these stages.


timestamp: 2025-05-14 20:42:55.725008
summary: Centralized AssetProcessingContext for Data Flow
tags: ["architecture", "data_management", "pipeline_design"]
rationale:
Using a dedicated AssetProcessingContext object to carry all data, configuration, and state for a single asset through the pipeline simplifies stage interfaces and centralizes information. It avoids passing numerous parameters between stages.


timestamp: 2025-05-14 20:42:55.767215
summary: Separate Orchestration from Engine Logic
tags: ["architecture", "separation_of_concerns", "processing_engine"]
rationale:
The ProcessingEngine acts as a high-level controller, initializing and invoking the PipelineOrchestrator. The PipelineOrchestrator is responsible for the detailed management of stage execution and the core processing loop. This separation improves organization.


timestamp: 2025-05-14 20:42:55.817951
summary: Core Item Processing Loop: Prepare, Process (Regular/Merged), Scale, Save
tags: ["pipeline_design", "item_processing", "orchestration"]
rationale:
The core logic for handling individual processing items (derived from FileRules or MergeTaskDefinitions) follows a sequence: 1. Prepare items (load data, initial checks). 2. Process (apply specific transformations for regular maps or execute merge tasks). 3. Scale (apply global scaling rules). 4. Save (generate output variants). This standardized loop is managed by the PipelineOrchestrator using dedicated stages.


timestamp: 2025-05-14 20:44:24.731191
summary: Pipeline Stage Data Flow: Shared Modifiable Context
tags: ["pipeline_design", "data_flow", "architecture_decision", "asset_processing_context"]
rationale:
The processing pipeline stages operate on a shared `AssetProcessingContext` object. Each stage's `execute` method receives this context, can read from it, and directly modify it or add new data. The modified context is then returned and passed to the next stage. This approach was chosen to allow stages to build upon the results of previous stages in a sequential manner and to provide a centralized way to manage the state of an asset's processing. It avoids complex data passing mechanisms between stages and allows for flexibility in how stages interact with the overall asset data.


timestamp: 2025-05-14 20:46:30.564079
tags: ["prediction", "architecture", "gui", "llm", "presets"]
summary:
Adopt a dual strategy for initial rule generation, supporting both deterministic Rule-Based (preset-driven) and adaptive LLM-Based prediction.

rationale:
Provides flexibility for users. Rule-based prediction offers speed, predictability, and explicit control via presets, suitable for well-defined naming conventions and known asset structures. LLM-based prediction offers an intelligent, adaptive approach for more ambiguous or novel asset structures where explicit rules might be complex to define or maintain. This dual approach caters to different user preferences and asset source complexities. Key components: [`RuleBasedPredictionHandler`](gui/prediction_handler.py:274), [`LLMPredictionHandler`](gui/llm_prediction_handler.py:22).


timestamp: 2025-05-14 20:46:35.613338
summary: Implement prediction handlers to run asynchronously in separate QThreads.
tags: ["prediction", "gui", "threading", "ux", "performance"]
rationale:
Prediction processes, especially LLM-based ones or rule-based ones on large file lists, can be time-consuming. Running them in separate `QThread`s prevents the GUI from freezing, ensuring a responsive user experience. The [`BasePredictionHandler`](gui/base_prediction_handler.py:25) manages the thread interaction and signaling. This allows users to continue interacting with the application while predictions are in progress.


timestamp: 2025-05-14 20:46:40.586030
summary: Define standardized signals in the base prediction handler for consistent communication.
tags: ["prediction", "gui", "signals", "architecture", "api-design"]
rationale:
The [`BasePredictionHandler`](gui/base_prediction_handler.py:25) defines common Qt signals: [`prediction_ready (str, list)`](gui/base_prediction_handler.py:33), [`prediction_error (str, str)`](gui/base_prediction_handler.py:37), and [`status_update (str)`](gui/base_prediction_handler.py:41). This ensures that all concrete prediction handlers (Rule-Based, LLM-Based) communicate their state (completion, errors, progress) to the main application in a uniform way, simplifying integration and event handling in the GUI.


timestamp: 2025-05-14 20:46:51.217320
tags: ["prediction", "llm", "prompt-engineering", "json", "api-design"]
summary:
Employ structured prompt engineering for LLM-based rule generation to ensure consistent and parsable JSON output.

rationale:
To reliably convert LLM responses into a [`SourceRule`](rule_structure.py:190) hierarchy, the [`LLMPredictionHandler`](gui/llm_prediction_handler.py:22) uses a detailed prompt template ([`_prepare_prompt`](gui/llm_prediction_handler.py:123)). This prompt includes system instructions, definitions for asset types and file types (with examples), example input/output pairs, and the list of files to analyze. The LLM is explicitly instructed to return a JSON object with a specific two-part schema (`individual_file_analysis` and `asset_group_classifications`). This structured approach, along with response sanitization ([`_parse_llm_response`](gui/llm_prediction_handler.py:217]), maximizes the chances of receiving valid, machine-readable output for rule creation.


timestamp: 2025-05-14 20:48:50.126597
summary: Utilize JSON as the exclusive format for all configuration files.
tags: ["configuration", "file_format", "json", "design_choice", "data_storage"]
rationale:
JSON was chosen for its human-readability, ease of parsing in Python (using the built-in `json` module), widespread support, and suitability for representing structured hierarchical data. This applies to core settings ([`config/app_settings.json`](config/app_settings.json:1)), user overrides ([`config/user_settings.json`](config/user_settings.json:1)), presets (`Presets/*.json`), type definitions ([`config/asset_type_definitions.json`](config/asset_type_definitions.json:1), [`config/file_type_definitions.json`](config/file_type_definitions.json:1)), supplier lists ([`config/suppliers.json`](config/suppliers.json:1)), and LLM settings ([`config/llm_settings.json`](config/llm_settings.json:1)). This consistency simplifies management and programmatic access via the [`Configuration`](configuration.py:93) class.


timestamp: 2025-05-14 20:48:56.532336
summary: Implement a hierarchical loading and merging strategy for configurations.
tags: ["configuration", "architecture", "merging_strategy", "presets", "overrides"]
rationale:
The system loads configurations in a specific order: core application settings ([`config/app_settings.json`](config/app_settings.json:1)) are loaded first, then user-specific settings ([`config/user_settings.json`](config/user_settings.json:1)) are deeply merged, allowing users to override default behaviors. Finally, a selected preset (`Presets/*.json`) provides context-specific rules and parameters. Centralized definitions ([`config/asset_type_definitions.json`](config/asset_type_definitions.json:1), [`config/file_type_definitions.json`](config/file_type_definitions.json:1)) are loaded to provide foundational data. This layered approach, managed by the [`Configuration`](configuration.py:93) class and its [`_deep_merge_dicts`](configuration.py:73) utility, offers flexibility by allowing global defaults, user-level personalization, and per-supplier/per-task customization through presets.


timestamp: 2025-05-14 20:49:04.499181
summary: Maintain centralized definitions for asset types and file types.
tags: ["configuration", "data_modeling", "type_system", "definitions", "consistency"]
rationale:
Asset type definitions (e.g., 'Surface', 'Model' including display color, description) are stored in [`config/asset_type_definitions.json`](config/asset_type_definitions.json:1), and file type definitions (e.g., 'MAP_COL', 'MAP_NRM' including bit-depth rules, keybinds, standard type identifiers) are in [`config/file_type_definitions.json`](config/file_type_definitions.json:1). This decision ensures consistency across the application (GUI, prediction logic, processing stages) when referring to or handling different kinds of assets and files. The [`Configuration`](configuration.py:93) class loads these ([`_load_asset_type_definitions`](configuration.py:266), [`_load_file_type_definitions`](configuration.py:286)) and makes them available, promoting maintainability as these core types can be updated in one place.


timestamp: 2025-05-14 20:49:15.088589
summary: Pre-compile regular expression patterns derived from preset configurations.
tags: ["configuration", "performance", "regex", "presets", "optimization", "initialization"]
rationale:
The [`Configuration`](configuration.py:93) class, in its [`_compile_regex_patterns`](configuration.py:140) method, converts fnmatch-style glob patterns and keyword lists from presets (e.g., `move_to_extra_patterns`, `map_type_mapping` keywords, `model_patterns`, `bit_depth_variants`) into compiled regular expression objects (`re.Pattern`). This is done once during initialization. This decision improves performance for subsequent file classification and rule application by avoiding repeated regex compilation during runtime, especially when processing many files or assets. The [`_fnmatch_to_regex`](configuration.py:33) utility assists in this conversion.


timestamp: 2025-05-14 20:50:24.074617
summary: Adopted a three-tier hierarchy (SourceRule, AssetRule, FileRule) for the dynamic rule system.
tags: ["hierarchical_rule_system", "design_decision", "architecture", "rule_structure"]
rationale:
A three-tier hierarchy provides a good balance between flexibility and complexity. It allows for global settings at the source level, asset-specific overrides, and fine-grained control at the individual file level. This structure supports diverse use cases, from batch processing of similar assets to highly customized handling of unique files, without becoming overly granular or difficult to manage. It also aligns well with the expected structure of input data (sources containing multiple assets, each with multiple files).


timestamp: 2025-05-14 20:53:24.513488
summary: Adopted PySide6 (Qt for Python) for GUI development and utilized the Model-View-Delegate pattern.
tags: ["gui", "pyside6", "qt", "model-view-delegate", "architecture_decision", "asset_processor_tool"]
rationale:
PySide6 was chosen for its comprehensive widget set, mature signals/slots mechanism for inter-component communication, and good cross-platform support. The Model-View-Delegate pattern (specifically with QAbstractItemModel, QTreeView, and custom delegates) was selected to effectively manage and display the complex hierarchical rule data (SourceRule -> AssetRule -> FileRule), separating data logic (UnifiedViewModel) from its presentation (QTreeView in MainPanelWidget) and editing (custom delegates). This promotes modularity and maintainability of the GUI codebase.


timestamp: 2025-05-14 20:54:42.282106
summary: Use PollingObserver from Watchdog library for file system monitoring.
tags: ["directory-monitor", "watchdog", "file-system-events", "compatibility", "polling"]
rationale:
The `PollingObserver` from the `watchdog` library was chosen over `InotifyObserver` (or other platform-specific observers) for better cross-platform compatibility, especially in environments like Docker or network shares where native event systems might be less reliable or require more complex configuration. While polling can be less efficient, its robustness in diverse environments is prioritized for the monitor system. See line 11 in monitor.py.


timestamp: 2025-05-14 20:54:47.358413
summary: Automated SourceRule generation for monitored assets.
tags: ["directory-monitor", "rule-generation", "automation", "prediction", "unattended-processing"]
rationale:
For assets processed via the Directory Monitor System, `SourceRule` objects are generated automatically by the `generate_source_rule_from_archive` function (from `utils.prediction_utils`). This approach is designed for unattended operation, bypassing the interactive rule creation available in the GUI. This implies that the prediction logic must be robust enough or rely on well-defined archive structures/presets to create effective rules without user intervention. See line 162 in monitor.py.


timestamp: 2025-05-14 20:54:51.879795
summary: Employ ThreadPoolExecutor for concurrent asset processing in monitor.
tags: ["directory-monitor", "concurrency", "threading", "performance", "ThreadPoolExecutor"]
rationale:
The Directory Monitor System uses a `concurrent.futures.ThreadPoolExecutor` to process multiple incoming archives concurrently. This allows the system to handle bursts of new files without blocking the main monitoring thread and improves throughput. The number of worker threads is configurable (defaulting to half CPU cores). See lines 10, 44-45, 78, 102-108 in monitor.py.


timestamp: 2025-05-14 20:57:51.925675
summary: Separated Blender node group creation and material creation into two distinct scripts.
rationale:
The Blender integration uses two separate scripts: [`create_nodegroups.py`](blenderscripts/create_nodegroups.py:1) to populate a library .blend file with PBRSET node groups, and [`create_materials.py`](blenderscripts/create_materials.py:1) to create materials in a project .blend file that link to these node groups. This separation allows for a centralized, reusable library of node groups that can be linked into multiple project files, promoting consistency and reducing duplication. Materials are created in the context of a project, referencing the shared node group library.

tags:
["blender", "architecture_decision", "script_design", "nodegroup_library", "material_creation", "asset_processor_tool"]


timestamp: 2025-05-14 20:59:58.519322
summary: Use argparse for CLI argument parsing.
tags: ["cli", "argparse", "main.py", "design_decision", "core_application_entry"]
rationale:
The `argparse` module from the Python standard library is used in `main.py` (see `setup_arg_parser()` function) to define and parse command-line arguments for the (currently non-functional) CLI mode. This provides a standard and robust way to handle CLI inputs.


timestamp: 2025-05-14 21:00:06.468763
summary: Use tempfile.mkdtemp for temporary workspace creation.
rationale:
The `utils.workspace_utils.prepare_processing_workspace()` function uses `tempfile.mkdtemp` to create unique temporary directories for processing assets. This is a standard Python library function that handles secure creation of temporary directories, reducing risks of conflicts or predictable names.

tags:
["temporary_files", "workspace_management", "tempfile", "utils_workspace_utils", "design_decision", "security"]


timestamp: 2025-05-14 21:00:13.041941
summary: Implement incrementing output directory values using glob and regex.
tags: ["path_utils", "output_naming", "incrementing_value", "glob", "regex", "design_decision"]
rationale:
The `utils.path_utils.get_next_incrementing_value()` function determines the next available integer for output directory naming (e.g., 'MyAsset_01', 'MyAsset_02'). It constructs a glob pattern to find existing matching directories and then uses regex to extract the numerical part, ensuring correct parsing and incrementing. This approach handles existing numbered directories and finds the next sequence value.


timestamp: 2025-05-14 21:00:18.418788
summary: Use QThreadPool and QRunnable for concurrent GUI-initiated processing tasks.
rationale:
The `App` class in `main.py` utilizes a `QThreadPool` to manage `ProcessingTask` (QRunnable) instances. This allows multiple asset processing operations, initiated from the GUI, to run concurrently in separate threads, preventing the GUI from freezing during potentially long-running operations. Each task handles its own workspace preparation, engine invocation, and cleanup.

tags:
["gui", "concurrency", "threading", "QThreadPool", "QRunnable", "main.py", "design_decision", "performance", "ux"]


timestamp: 2025-05-14 21:10:33.072505
summary: ProcessingItem.image_data is transient and not part of persisted rule structure.
rationale:
Code review of rule_structure.py confirms that ProcessingItem.image_data (np.ndarray) is explicitly excluded from pickling via __getstate__ and is not handled in JSON serialization methods (to_json/from_json). This design reflects that rules define *how* to process, while ProcessingItems carry image data *during* runtime processing, avoiding serialization complexities and large JSON files.


timestamp: 2025-05-14 21:11:42.159462
summary: Monitor: Categorized Error Directory Naming
tags: ["monitor", "error-handling", "file-management"]
rationale:
The monitor categorizes processing errors by their exception type and uses the lowercase exception class name as the reason for moving the failed archive to the error directory. This provides immediate context in the error directory structure about *why* a specific archive failed processing (e.g., `configurationerror`, `predictionerror`, `workspaceerror`, `processingerror`, `file_not_found`, `unexpected_exception`).


timestamp: 2025-05-14 21:11:45.317811
summary: Monitor: Explicit Temporary Workspace Management
tags: ["monitor", "workspace-management", "cleanup"]
rationale:
The monitor explicitly manages temporary processing workspaces for each archive by using `prepare_processing_workspace` for creation/extraction and `shutil.rmtree` for cleanup in a `finally` block. This ensures a clean environment for processing each archive and guarantees cleanup even if errors occur, preventing accumulation of temporary files.


timestamp: 2025-05-14 21:11:48.346915
summary: Monitor: Output Path Hashing and Incrementation
tags: ["monitor", "output-naming", "hashing", "incrementation"]
rationale:
The monitor incorporates SHA5 hashes of the input archive and an incrementing value (if configured in the output pattern) into the parameters passed to the `ProcessingEngine` for output path naming. Including the SHA5 provides a unique identifier derived from the input content, aiding in tracking and preventing duplicate processing. The incrementing value allows for sequential numbering of output batches from the same source if needed.


timestamp: 2025-05-14 21:11:51.387687
summary: Monitor: Environment Variable Driven Configuration
tags: ["monitor", "configuration", "environment-variables", "deployment"]
rationale:
The monitor's key operational parameters (input/output directories, polling interval, log level, etc.) are configured primarily through environment variables. This design choice enhances flexibility, particularly for deployment scenarios (like Docker), allowing easy configuration changes without modifying code or requiring separate configuration files within the application itself.


timestamp: 2025-05-14 21:11:55.364611
summary: Monitor: Fallback for Optional `workspace_utils` Import
tags: ["monitor", "imports", "resilience", "modularity"]
rationale:
The monitor includes a try-except block for importing `utils.workspace_utils`, providing fallback dummy functions if the import fails. This makes the monitor more resilient to environments where `workspace_utils` might not be available or correctly installed, potentially allowing basic monitoring to continue (though processing tasks requiring workspace management would fail gracefully). It might also support development workflows where this utility is optional.


timestamp: 2025-05-14 21:12:32.524253
summary: Blender Scripts: Reliance on Specific Template Names
tags: ["Blender Integration", "Scripting", "Templates", "Node Groups", "Materials"]
rationale:
The Blender scripts `create_nodegroups.py` and `create_materials.py` rely on specific, hardcoded template names for node groups and materials (`Template_PBRSET`, `Template_PBRTYPE`, `Template_PBRMaterial`). They also expect specific placeholder node labels within these templates (e.g., `PBRSET_PLACEHOLDER`, `AspectRatioCorrection`, `Histogram-*`, `HighestResolution`, and map type names like `COL`, `NRM`). This convention is essential for the scripts to find and modify the correct nodes and structures when creating asset-specific node groups and materials.


timestamp: 2025-05-14 21:12:36.428227
summary: Blender Add-on: Assumed Node Groups for Material Merger
tags: ["Blender Integration", "Add-on", "Material Merger", "Node Groups", "Assumptions"]
rationale:
The `MaterialMerge` add-on (`blender_addon/material_merger/operator.py`) includes logic to handle both already-merged materials and base materials. When processing a source material that is *not* a top-level merged material, it assumes the presence of specific node groups named `PBR_Handler` and `PBR_BSDF` within that material's node tree. These are expected to provide the 'Displacement' and 'BSDF' outputs, respectively, for the merging process.


timestamp: 2025-05-14 21:12:40.191247
summary: Blender Scripts: Base64 Encoding for Child Node Group Names
tags: ["Blender Integration", "Scripting", "Node Groups", "Naming Conventions", "Base64"]
rationale:
The `create_nodegroups.py` script uses Base64 encoding for the names of the child node groups it creates (those representing individual map types like COL, NRM, etc., within the main PBRSET group). This is implemented in the `encode_name_b64` function and applied when setting the `name` property of the new child node groups. This approach is likely used to ensure that the generated node group names are valid within Blender's naming conventions, especially if the original map type strings contain characters that Blender might not handle well.


timestamp: 2025-05-14 21:12:43.734717
summary: Blender Scripts: Command-Line Argument for Path Overrides
tags: ["Blender Integration", "Scripting", "Command Line Arguments", "Configuration"]
rationale:
Both `create_nodegroups.py` and `create_materials.py` Blender scripts are designed to accept command-line arguments passed after a '--' separator. This mechanism is used to override the default paths specified within the scripts, specifically the root directory of the processed asset library (`PROCESSED_ASSET_LIBRARY_ROOT`). `create_materials.py` also accepts a second argument to override the path to the .blend file containing the PBRSET node groups (`NODEGROUP_BLEND_FILE_PATH`). This provides flexibility when running the scripts in different environments or with different configurations.


timestamp: 2025-05-14 21:13:10.278311
summary: GUI: Per-Task Configuration and Engine Instantiation
rationale:
The ProcessingTask in main.py instantiates its own Configuration and ProcessingEngine for each task based on the SourceRule's preset. This ensures task isolation with specific preset configurations when processing is initiated from the GUI.


timestamp: 2025-05-14 21:14:48.309550
summary: Workspace Utils: Initial Archive Support Limited to ZIP
rationale:
Based on the code review of utils/workspace_utils.py, the `prepare_processing_workspace` function and `SUPPORTED_ARCHIVES` variable explicitly show support only for .zip files for archive extraction.


timestamp: 2025-05-15 17:00:32.445539
summary: Adopt 'Explicit Prioritization via Enhanced Variant Mapping' for 16-bit map type prioritization.
tags: ["architecture", "preset", "prioritization", "map_type_mapping", "item_type", "16-bit_variant"]
rationale:
This approach (Solution 2 from the architectural assessment) is preferred for its robustness, clarity, maintainability, and user control. It involves introducing a new `prioritized_variants_map` list in the `source_naming` section of preset files. This list will explicitly define prioritized variant patterns (e.g., `*_NRM16*`) and their target `item_type`s (e.g., `MAP_NRM16`), ensuring they take precedence over more general matches from `map_type_mapping`. Problem Addressed: Corrects the issue where `_DISP16_` and `_NRM16_` map types do not take priority over their non-16-bit counterparts, which causes [`autotest.py`](autotest.py:1) failures. The root cause was identified as keyword order dependency in `map_type_mapping` and ineffective `bit_depth_variants` for `item_type` prioritization.


timestamp: 2025-05-15 17:24:14.504842
summary:
Adopt user-refined approach for map type prioritization: embed `priority_keywords` within `map_type_mapping` entries.

rationale:
This approach, preferred by the user, co-locates all keywords for a conceptual map type. If a file matches a `priority_keyword` in a `map_type_mapping` entry, it gets the `target_type`, and other files matching only regular `keywords` of that *same entry* are ignored (e.g., `item_type` = "FILE_IGNORE"). This directly handles cases like using a 16-bit map variant and discarding its 8-bit counterpart if both are supplied for the same conceptual map. While increasing prediction logic complexity, it offers a user-friendly preset structure. This refines ConPort Decision ID 36.

tags:
["architecture", "preset", "prioritization", "map_type_mapping", "item_type", "16-bit_variant", "refinement", "FILE_IGNORE"]


timestamp: 2025-05-15 19:32:52.156816
tags: ["configuration", "first_time_setup", "portability", "user_experience", "compilation_plan"]
summary:
Default Configuration Location Strategy: Prioritize portable 'user_data/' next to EXE, fallback to user profile app data if not writable, always allow custom folder choice.

rationale:
Balances portability with usability and handles permissions issues for typical installations. Aligns with user preference for a portable default while providing robust fallbacks and user control.


timestamp: 2025-05-15 19:32:59.980867
tags: ["configuration", "first_time_setup", "data_preservation", "user_experience", "compilation_plan"]
summary:
Handling Existing User Configs (First-Time Setup): Preserve existing user files; bundled defaults are only copied if corresponding file is missing in user's target location.

rationale:
Protects existing user customizations and data. Ensures that a user's pre-existing setup or partially completed setup is not inadvertently overwritten by default files during the initial configuration process. `user_settings.json` is loaded if present, or created new based on UI input if not.


timestamp: 2025-05-15 19:33:07.725108
tags: ["configuration", "error_handling", "resilience", "compilation_plan"]
summary:
Fallback for Missing User-Modifiable Configs (Post First-Time Setup): Restore missing modifiable files (excluding user_settings.json) from bundle defaults.

rationale:
Ensures application stability and provides a recovery path if a user accidentally deletes essential (but modifiable) configuration files from their established config directory after the initial setup. A warning will be logged, and user notification considered.


timestamp: 2025-05-16 07:15:11.899213
summary: GUI Drag+Drop Regression - Initial Cause (Missing Signal Emission)
tags: ["gui", "drag-and-drop", "regression", "debug", "unified-view-model", "asset-restructure-handler"]
rationale:
Investigation revealed that the UnifiedViewModel.dropMimeData method was not emitting the targetAssetOverrideChanged signal after a drag-and-drop move, preventing the AssetRestructureHandler from performing necessary restructuring and cleanup.


timestamp: 2025-05-16 07:15:18.087179
summary: GUI Drag+Drop Regression - Serialization Error (QPersistentModelIndex)
tags: ["gui", "drag-and-drop", "regression", "debug", "unified-view-model", "serialization", "pyside6"]
rationale:
Attempting to serialize QPersistentModelIndex directly with QDataStream in UnifiedViewModel.mimeData caused an AttributeError. The fix involved changing the drag-and-drop data transfer to use file paths (strings) instead of QPersistentModelIndex, and updating dropMimeData to find items by path.


timestamp: 2025-05-16 07:15:23.943858
summary: GUI Drag+Drop Regression - Unhashable Type Error (AssetRule in Set)
tags: ["gui", "drag-and-drop", "regression", "debug", "unified-view-model", "type-error", "hashable"]
rationale:
The TypeError: unhashable type: 'AssetRule' occurred because AssetRule objects were added to a set (original_parents_to_check). The fix involved storing a hashable representation (tuple of source path and asset name) instead of the AssetRule object itself, and updating the cleanup logic to retrieve the object by these identifiers.


timestamp: 2025-05-16 08:09:54.958122
summary: Completed codebase search for old bit depth terminology (Phase 5).
tags: ["refactoring", "bit-depth", "phase-5", "search"]
rationale:
Identified remaining usages of 'bit_depth_rule', 'output_bit_depth', and related 'respect' values in Python and JSON files, excluding previously refactored files. Findings are listed in the subsequent attempt_completion.


timestamp: 2025-05-16 08:21:34.939193
summary: Resolved persistent bit depth terminology warnings in autotest.py
tags: ["debug", "bit-depth", "configuration", "pipeline"]
rationale:
Investigation revealed that the bit depth terminology migration logic in configuration.py was incorrectly placed, preventing it from being applied to file type definitions loaded from bundled defaults during autotest. Additionally, a mismatch in map type field names between pipeline dataclasses (ProcessedRegularMapData and SaveVariantsInput) and stages (Orchestrator and SaveVariantsStage) caused the correct bit_depth_policy to be inaccessible in image_saving_utils.py.


timestamp: 2025-05-16 08:46:39.510545
summary: Refactored bit depth terminology
tags: ["refactoring", "bit-depth", "configuration", "image-processing"]
rationale:
Unified inconsistent terminology ('bit_depth_rule', 'output_bit_depth', 'respect', 'respect_inputs') to a single key ('bit_depth_policy') with clear values ('force_8bit', 'force_16bit', 'preserve', ''). Corrected logic in image saving utility and configuration loading.


timestamp: 2025-05-16 08:46:39.567137
summary: Implemented bit depth terminology migration logic
tags: ["refactoring", "bit-depth", "configuration", "migration", "backward-compatibility"]
rationale:
Added logic to Configuration class (_load_definition_file_with_fallback) to automatically convert old 'bit_depth_rule' key and 'respect' value to 'bit_depth_policy' and 'preserve' when loading older file_type_definitions.json files, ensuring backward compatibility.


timestamp: 2025-05-16 08:46:39.630584
summary: Resolved persistent autotest.py bit depth warnings
tags: ["debugging", "autotest", "bit-depth", "configuration", "pipeline"]
rationale:
Investigated and fixed issues causing 'Unknown or empty bit_depth_policy' warnings in autotest.py. Involved correcting Configuration object usage, populating SaveVariantsInput with migrated data, and using correct map type identifiers in image_saving_utils.py. Also addressed handling of merged map types.


# progress_entries

timestamp: 2025-05-14 21:14:39.282928
status: Open
description:
The `run_cli()` function is called in main.py but is not defined, indicating incomplete CLI functionality.


timestamp: 2025-05-15 22:01:07.569131
status: TODO
description:
Complete remaining Asset Processor compilation plan: Cythonization, PyInstaller packaging, full testing, and documentation (as per compilation_assessment_plan.md).


timestamp: 2025-05-16 07:15:29.804934
status: DONE
description:
Investigated and fixed the regression where drag+drop functionality was broken after the GUI file table refactor. Issues included missing signal emission, QPersistentModelIndex serialization error, and unhashable AssetRule in set. Implemented fixes in gui/unified_view_model.py.


# system_patterns

name: Centralized Configuration Management
tags: ["configuration", "core_component"]
description:
Manages loading, merging, and providing access to all application settings, including core configurations, presets, user overrides, and type definitions. Represented by the Configuration class in configuration.py.


name: Main Application Orchestrator
tags: ["application_flow", "core_component", "gui_management"]
description:
Acts as the central coordinator for the application, managing the GUI, handling user interactions, initiating processing tasks, and connecting various components. Represented by the App class in main.py.


name: Qt Model-View-Delegate Architecture
tags: ["gui", "architecture_pattern", "qt", "model_view"]
description:
The GUI extensively uses Qt's Model-View-Delegate pattern. Key components include UnifiedViewModel, PreviewTableModel, various custom delegates in delegates.py, and QTreeView / QTableView widgets. This pattern separates data representation from its presentation and editing. Supporting Files: gui/unified_view_model.py, gui/preview_table_model.py, gui/delegates.py, gui/main_panel_widget.py.


name: Dedicated Configuration Editors UI
tags: ["gui", "configuration_management", "ui_pattern"]
description:
The application provides distinct, specialized dialogs and widgets for managing different configuration aspects: Application Preferences (ConfigEditorDialog for user_settings.json), Core Definitions (DefinitionsEditorDialog for asset types, file types, suppliers), Presets (PresetEditorWidget for preset .json files), and LLM Settings (LLMEditorWidget for llm_settings.json). Supporting Files: gui/config_editor_dialog.py, gui/definitions_editor_dialog.py, gui/preset_editor_widget.py, gui/llm_editor_widget.py.


name: Asynchronous Task Handling (Prediction & Processing)
tags: ["gui", "asynchronous_processing", "threading", "performance"]
description:
Long-running tasks like rule-based prediction, LLM prediction, and asset processing are handled asynchronously to keep the GUI responsive. Prediction tasks (RuleBasedPredictionHandler, LLMPredictionHandler) run as QRunnable in a QThreadPool. LLMInteractionHandler coordinates LLM communication. Asset processing (main.ProcessingTask) is also initiated by the GUI but runs in a separate process. Supporting Files: gui/base_prediction_handler.py, gui/prediction_handler.py, gui/llm_prediction_handler.py, gui/llm_interaction_handler.py, main.py.


name: Staged Processing Pipeline Engine
tags: ["processing", "core_component", "pipeline", "orchestration"]
description:
Orchestrates asset processing through a series of discrete, modular stages. Key component is the PipelineOrchestrator (processing/pipeline/orchestrator.py) which manages the execution of Pre-Item, Core Item Loop, and Post-Item stages for each asset. Each stage performs a specific task on the asset data held in an AssetProcessingContext. Represented initially by the ProcessingEngine class (processing_engine.py) which sets up the orchestrator.


name: Post-Processing Blender Integration
tags: ["blender", "integration", "post_processing", "automation", "materials", "nodegroups"]
description:
Automates the creation and configuration of Blender materials and node groups for assets processed by the tool. Scripts in the 'blenderscripts/' directory (create_materials.py, create_nodegroups.py) are run within Blender. They read the 'metadata.json' of processed assets, link to or create PBR node setups, and connect the processed texture maps to the appropriate material inputs. This facilitates immediate use of processed assets in Blender.


name: Automated End-to-End Rule & Processing Verification
tags: ["testing", "automation", "e2e_test", "rule_engine_validation", "qt_interaction"]
description:
autotest.py implements an automated test that simulates GUI interactions to load an asset, apply a preset, verify the generated rule hierarchy against an expected JSON definition, trigger asset processing, and check for basic output and log integrity. It uses Qt's event loop to handle asynchronous operations within the main application. Key Components: AutoTester class, expected rules JSON file, CLI for test parameters.


name: Processing Engine & Pipeline Orchestration
tags: ["architecture", "pipeline", "orchestration", "core_processing"]
description:
Manages the overall asset processing workflow. It initializes the processing context for each asset, sequences through discrete pre-processing, item-specific processing (including preparation, scaling, and saving), and post-processing stages. It handles the flow of data and state via an AssetProcessingContext. Key components include the ProcessingEngine (top-level controller), PipelineOrchestrator (detailed sequence manager), and AssetProcessingContext (data carrier). Key files: processing_engine.py, processing/pipeline/orchestrator.py, processing/pipeline/asset_context.py.


name: Modular Pipeline Stage
tags: ["pipeline", "asset_processing", "modularity", "design_pattern"]
description:
Defines a pattern where individual processing steps in an asset pipeline are encapsulated as distinct, interchangeable stages. Each stage inherits from a common base (e.g., `ProcessingStage`), implements a core `execute(context)` method, and operates on a shared `AssetProcessingContext`. This promotes modularity, reusability, and easier maintenance of the processing workflow.


name: Prediction Handler Core
tags: ["prediction", "core", "gui", "asynchronous"]
description:
Provides the foundational abstract base class ([`BasePredictionHandler`](gui/base_prediction_handler.py:25)) for all prediction systems that generate `SourceRule` hierarchies. It manages asynchronous execution in a `QThread`, defines standardized signals for completion ([`prediction_ready`](gui/base_prediction_handler.py:33)), errors ([`prediction_error`](gui/base_prediction_handler.py:37)), and status updates ([`status_update`](gui/base_prediction_handler.py:41)). Concrete handlers implement the [`_perform_prediction`](gui/base_prediction_handler.py:121) method for their specific logic and can be cancelled via the [`cancel()`](gui/base_prediction_handler.py:111) slot. Key file: [`gui/base_prediction_handler.py`](gui/base_prediction_handler.py:1)


name: Rule-Based Prediction Handler
tags: ["prediction", "rule-based", "presets", "regex", "gui"]
description:
Implements [`RuleBasedPredictionHandler`](gui/prediction_handler.py:274) for generating [`SourceRule`](rule_structure.py:190) hierarchies based on predefined presets and file naming conventions. It uses a `Configuration` object loaded from a preset to drive the [`classify_files`](gui/prediction_handler.py:40) function. This function employs a two-pass regex-based system to identify prioritized bit-depth variants, general maps, and extra files. It then constructs the `SourceRule`, `AssetRule`, and `FileRule` objects, predicting asset types (Model, Decal, Surface) based on file patterns and keywords. Key file: [`gui/prediction_handler.py`](gui/prediction_handler.py:1)


name: LLM-Based Prediction Handler
tags: ["prediction", "llm", "ai", "gui", "api-integration"]
description:
Implements [`LLMPredictionHandler`](gui/llm_prediction_handler.py:22) for generating [`SourceRule`](rule_structure.py:190) hierarchies by interacting with a Large Language Model. It prepares a detailed prompt ([`_prepare_prompt`](gui/llm_prediction_handler.py:123)) including asset/file type definitions, examples, and a list of relative file paths. It calls the LLM API ([`_call_llm`](gui/llm_prediction_handler.py:155)) and parses the expected two-part JSON response ([`_parse_llm_response`](gui/llm_prediction_handler.py:217)) containing individual file analysis and asset group classifications. Includes response sanitization for comments and markdown. Key file: [`gui/llm_prediction_handler.py`](gui/llm_prediction_handler.py:1)


name: Configuration System
tags: ["configuration", "core_component", "json", "presets", "settings_management"]
description:
Loads, manages, and provides access to all application settings, supplier-specific presets, and centralized type definitions. It merges core settings ([`config/app_settings.json`](config/app_settings.json:1)), user overrides ([`config/user_settings.json`](config/user_settings.json:1)), a selected preset ([`Presets/*.json`](Presets/*.json:1)), asset type definitions ([`config/asset_type_definitions.json`](config/asset_type_definitions.json:1)), file type definitions ([`config/file_type_definitions.json`](config/file_type_definitions.json:1)), supplier configurations ([`config/suppliers.json`](config/suppliers.json:1)), and LLM settings ([`config/llm_settings.json`](config/llm_settings.json:1)) to provide a consistent configuration state. Key class: [`Configuration`](configuration.py:93) in [`configuration.py`](configuration.py:1).


name: Hierarchical Rule System
tags: ["rules", "processing_control", "dynamic_configuration", "hierarchy"]
description:
A dynamic, layered, and fine-grained mechanism for controlling processing parameters. It uses a three-tier hierarchy (SourceRule, AssetRule, FileRule) to define processing instructions that can override static presets. Populated by the Prediction System and consumed by the Processing Engine.


name: Main GUI Architecture (Asset Processor)
tags: ["gui", "architecture", "pyside6", "model-view", "asset_processor_tool"]
description:
The Graphical User Interface (GUI) is built using PySide6 and organized around a central MainWindow. It features a MainPanelWidget for primary user interaction, displaying a hierarchical view of asset processing rules (SourceRule -> AssetRule -> FileRule) via a UnifiedViewModel (QAbstractItemModel). Specialized editor widgets (PresetEditorWidget, LLMEditorWidget) and dialogs (DefinitionsEditorDialog, ConfigEditorDialog) handle configuration. Asynchronous operations (prediction, processing) are managed via QThreads and dedicated handlers (RuleBasedPredictionHandler, LLMInteractionHandler) to maintain UI responsiveness. Key files: gui/main_window.py, gui/main_panel_widget.py, gui/unified_view_model.py.


name: GUI: Unified View Model & Hierarchical Data Handling
description:
The GUI employs a UnifiedViewModel (QAbstractItemModel) to manage and present the hierarchical SourceRule -> AssetRule -> FileRule structure. This model provides data for display (including dynamic background/foreground colors based on item types from configuration), handles editing via setData(), supports drag-and-drop of FileRules between AssetRules, and merges new prediction data while preserving user overrides. It uses custom delegates for specialized column editing. Key file: gui/unified_view_model.py. Signals like targetAssetOverrideChanged and assetNameChanged facilitate communication with other GUI components for data synchronization and restructuring.

tags:
["gui", "qt_model_view", "data_management", "hierarchical_data", "asset_processor_tool", "unified_view_model"]


name: Directory Monitor System
description:
Continuously watches a specified directory (INPUT_DIR) for new archive files (.zip, .rar, .7z). Upon detection, it waits for a brief configurable delay (PROCESS_DELAY), then automatically generates processing rules using `generate_source_rule_from_archive` and invokes the `ProcessingEngine` via a ThreadPoolExecutor to process the new assets. Uses the `watchdog` library (PollingObserver) for file system event monitoring. Key file: monitor.py.

tags:
["automation", "file-monitoring", "unattended-processing", "watchdog", "rule-generation", "concurrent-processing"]


name: Blender Node Group Generation System
tags: ["blender", "nodegroup_generation", "automation", "python_script", "asset_processor_tool"]
description:
Blender script ([`blenderscripts/create_nodegroups.py`](blenderscripts/create_nodegroups.py:1)) that scans an Asset Processor Tool library, reads `metadata.json` files, and creates/updates PBRSET (parent) and PBRTYPE (child) node groups in the active Blender file. It loads textures, sets color spaces, aspect ratios, and other parameters based on metadata, using `Template_PBRSET` and `Template_PBRTYPE` node groups.


name: Blender Material Creation System
description:
Blender script ([`blenderscripts/create_materials.py`](blenderscripts/create_materials.py:1)) that reads `metadata.json` files from an Asset Processor Tool library. It creates new Blender materials by copying a `Template_PBRMaterial` and links pre-existing PBRSET node groups (expected to be in a separate library .blend file, identified by `PBRSET_AssetName`) into a placeholder node within the new material. It also sets material previews and viewport display properties based on metadata.

tags:
["blender", "material_creation", "automation", "python_script", "asset_processor_tool", "nodegroup_linking"]


name: Blender Material Merger Add-on
tags: ["blender", "material_merging", "addon", "ui", "nodegraph_manipulation", "asset_processor_tool"]
description:
A Blender add-on ([`blender_addon/material_merger/`](blender_addon/material_merger/)) that provides a UI panel and an operator to merge two existing materials. It copies the node trees of the source materials into a new material and uses a dedicated `MaterialMerge` node group (from [`blender_addon/material_merger/blender_files/utility_nodegroups.blend`](blender_addon/material_merger/blender_files/utility_nodegroups.blend:1)) to combine their BSDF and Displacement outputs. Designed to work with materials potentially structured with `PBR_Handler` and `PBR_BSDF` node groups or already merged materials.


name: Core Utilities Collection
description:
A collection of shared, reusable utility functions for common tasks across the Asset Processor Tool. This includes file and path manipulation (generating paths from patterns, sanitizing filenames, getting next incrementing values for output directories), temporary workspace management (creating and preparing temporary directories from archives or folders), and file hashing (SHA256 calculation).

tags:
["utilities", "file_operations", "path_manipulation", "workspace_management", "hashing", "core_component"]


name: Application Entry & GUI Task Management
description:
Handles the main application startup, distinguishing between GUI and CLI modes. For GUI mode, it initializes the PySide6 application, loads configuration, and sets up the main window and processing engine. A key part is the App class which orchestrates GUI-initiated processing tasks using a QThreadPool and QRunnable (ProcessingTask) for concurrent execution. This includes preparing workspaces, calculating hashes/incrementing values, and invoking the ProcessingEngine for each asset source. The (currently non-functional) CLI entry point is also managed here via argparse.

tags:
["application_entry", "gui_management", "cli_handling", "task_management", "concurrency", "qthreadpool", "core_component", "pyside6"]


name: SupplierDeterminationStage
tags: ["PipelineStage", "AssetIngestion", "Metadata"]
description:
A pipeline stage responsible for determining the original supplier or source of an asset. It typically analyzes file names, folder structures, or embedded metadata against a predefined list of known suppliers and their conventions. This allows for supplier-specific processing rules or metadata tagging later in the pipeline.


name: FileRuleFilterStage
tags: ["PipelineStage", "Filtering", "FileProcessing"]
description:
A pipeline stage that filters incoming files based on predefined rules. These rules typically involve file naming conventions, extensions, or other metadata to determine which files are relevant for further processing and which should be ignored or handled differently.


name: InitialScalingStage
tags: ["PipelineStage", "ImageProcessing", "Scaling"]
description:
A pipeline stage responsible for performing initial scaling or resizing operations on input images. This is often done early in the pipeline to standardize image dimensions or reduce processing time for subsequent stages. The scaling mode and target dimensions are typically determined by configuration or asset-specific rules.


name: OutputOrganizationStage
tags: ["PipelineStage", "Output", "FileOrganization"]
description:
A pipeline stage responsible for organizing the processed output files into a structured directory layout. This involves determining final file paths, creating necessary directories, and ensuring the output adheres to the desired structure based on configuration or asset type.


name: MetadataInitializationStage
tags: ["PipelineStage", "Metadata", "AssetIngestion"]
description:
A pipeline stage responsible for initializing or extracting metadata associated with an asset. This can involve reading existing metadata from source files, generating initial metadata based on file properties, or setting up a data structure to hold metadata that will be enriched throughout the pipeline.


name: AlphaExtractionToMaskStage
tags: ["PipelineStage", "ImageProcessing", "AlphaChannel", "Masking"]
description:
A pipeline stage that extracts the alpha channel from an image and converts it into a separate mask image. This is useful for assets where transparency or masking information is stored in the alpha channel of a color map.


name: AssetSkipLogicStage
tags: ["PipelineStage", "Filtering", "Optimization"]
description:
A pipeline stage that implements logic to determine if an entire asset or specific processing tasks for an asset should be skipped based on predefined criteria, such as file existence, timestamps, or user settings.


name: GlossToRoughConversionStage
tags: ["PipelineStage", "ImageProcessing", "PBR", "MaterialConversion"]
description:
A pipeline stage that converts a glossiness map to a roughness map. This is a common step in PBR workflows, as different rendering engines use either glossiness or roughness to define surface reflectivity.


name: NormalMapGreenChannelStage
tags: ["PipelineStage", "ImageProcessing", "NormalMaps", "PBR"]
description:
A pipeline stage that handles the green channel convention for normal maps. Different software and engines use either a +Y or -Y convention for the green channel, and this stage ensures compatibility by potentially inverting the green channel.


name: PrepareProcessingItemsStage
tags: ["PipelineStage", "TaskPreparation", "Workflow"]
description:
A pipeline stage responsible for preparing the individual items or tasks that will be processed in subsequent stages. This might involve identifying specific files to process, determining the required operations for each file, and creating a structured list of processing tasks.


name: RegularMapProcessorStage
tags: ["PipelineStage", "ImageProcessing", "TextureMaps"]
description:
A pipeline stage dedicated to processing 'regular' texture maps (e.g., diffuse, albedo, metallic) that don't require specialized handling like normal maps or masks. This stage might involve color space conversions, format changes, or basic adjustments.


name: SaveVariantsStage
tags: ["PipelineStage", "Output", "Variants"]
description:
A pipeline stage responsible for saving different variants or versions of the processed assets. This could include saving at different resolutions, formats, or with different channel packing based on configuration or preset settings.


name: Automated Rule Generation (Filename Trigger)
tags: ["rule-generation", "automation", "directory-monitor", "system-pattern"]
description:
This system pattern describes the automated process of generating Source Rules based on the input archive filename. The `generate_source_rule_from_archive` function in `utils/prediction_utils.py` extracts a preset name from the filename (e.g., `[PresetName]_...`) and loads the corresponding preset to build the initial Source Rule hierarchy. This process is typically used by automated workflows like the Directory Monitor and is distinct from the interactive, GUI-based prediction methods (Rule-Based and LLM-based) which require explicit user input or selection.


name: GUI: Custom Editor Delegates for TreeView
tags: ["GUI", "Qt", "TreeView", "Delegates", "Editing"]
description:
The application utilizes custom QStyledItemDelegate implementations (e.g., ComboBoxDelegate, LineEditDelegate) in conjunction with the UnifiedViewModel to provide specialized in-cell editors directly within the QTreeView. This allows users to edit properties like Asset Type, Item Type, and Target Asset using appropriate widgets (like dropdowns or line edits) for a more interactive and efficient editing experience. The delegates access data and available options from the UnifiedViewModel.

References:
- gui/delegates.py
- gui/unified_view_model.py


name: GUI: Reactive Asset Restructuring Logic
tags: ["GUI", "Qt", "Model", "Restructuring", "Handler"]
description:
The AssetRestructureHandler is a key component in the GUI's reactive architecture. It connects to signals from the UnifiedViewModel, specifically `targetAssetOverrideChanged` and `assetNameChanged`. Its primary function is to automatically restructure the hierarchical model when a user changes a FileRule's target asset override or renames an AssetRule. When a FileRule's target asset override is changed, the handler orchestrates the movement of that FileRule to the specified target AssetRule, creating the target AssetRule if it doesn't exist, and cleaning up the original parent AssetRule if it becomes empty. This ensures the model's structure remains consistent with user edits.

References:
- gui/asset_restructure_handler.py
- gui/unified_view_model.py
- gui/main_window.py (for connection)


# custom_data

category: ProjectGlossary
key: AssetProcessingContext
value:
{"definition": "The primary stateful data container within the asset processing pipeline. An instance is created for each AssetRule and passed sequentially through all processing stages, carrying all relevant data including source files, rules, configuration, intermediate results, and metadata.", "related_components": ["processing/pipeline/asset_context.py", "processing/pipeline/orchestrator.py", "Staged Processing Pipeline Engine (System Pattern ID: 7)"], "type": "Data Structure (Dataclass)"}


category: ProjectGlossary
key: SupplierDeterminationStage
value:
{"definition": "Determines the effective supplier for the asset based on source rules and configuration. (processing/pipeline/stages/supplier_determination.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: AssetSkipLogicStage
value:
{"definition": "Checks conditions to determine if the entire asset processing should be skipped. (processing/pipeline/stages/asset_skip_logic.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: MetadataInitializationStage
value:
{"definition": "Initializes the primary metadata structure for the asset being processed. (processing/pipeline/stages/metadata_initialization.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: FileRuleFilterStage
value:
{"definition": "Filters the asset's file rules to exclude ignored files and determine which ones proceed to processing. (processing/pipeline/stages/file_rule_filter.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: GlossToRoughConversionStage
value:
{"definition": "Converts glossiness maps to roughness maps by inverting image data. (processing/pipeline/stages/gloss_to_rough_conversion.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: AlphaExtractionToMaskStage
value:
{"definition": "Extracts an alpha channel from a suitable source map to create a mask map if one isn't explicitly provided. (processing/pipeline/stages/alpha_extraction_to_mask.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: NormalMapGreenChannelStage
value:
{"definition": "Inverts the green channel of normal maps based on global or supplier-specific configuration. (processing/pipeline/stages/normal_map_green_channel.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: PrepareProcessingItemsStage
value:
{"definition": "Generates individual ProcessingItem objects for each map variant (e.g., different resolutions) and defines MergeTaskDefinitions. (processing/pipeline/stages/prepare_processing_items.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: InitialScalingStage
value:
{"definition": "Performs Power-of-Two (POT) scaling on image data for each processing item or merged map. (processing/pipeline/stages/initial_scaling.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: MergedTaskProcessorStage
value:
{"definition": "Executes defined map merging tasks, combining channels from multiple source maps into a single output map. (processing/pipeline/stages/merged_task_processor.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: SaveVariantsStage
value:
{"definition": "Saves the processed image data for each map variant (e.g., 1K, PREVIEW, LOWRES) to temporary files. (processing/pipeline/stages/save_variants.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: OutputOrganizationStage
value:
{"definition": "Arranges all processed temporary files into their final, structured output directories according to naming patterns. (processing/pipeline/stages/output_organization.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: MetadataFinalizationAndSaveStage
value:
{"definition": "Completes the asset's metadata, including final map paths, and saves it to a metadata.json file. (processing/pipeline/stages/metadata_finalization_save.py)", "type": "Processing Stage"}


category: ProjectGlossary
key: config/app_settings.json
value:
{"definition": "Core application settings including default output patterns, image processing parameters (resolutions, compression), map merging rules, Blender integration paths, temporary directory prefixes, global feature flags (e.g., low-resolution fallback), and general settings like global normal map inversion. Loaded by the Configuration class.", "type": "Configuration File"}


category: ProjectGlossary
key: config/asset_type_definitions.json
value:
{"definition": "Defines categories for assets (e.g., Surface, Model, Decal) along with their display color, description, and filename examples. Used by the GUI for display and color-coding, by the DefinitionsEditorDialog for management, and potentially by prediction logic.", "type": "Configuration File"}


category: ProjectGlossary
key: config/file_type_definitions.json
value:
{"definition": "Defines various file types, especially texture maps (e.g., MAP_COL, MAP_NRM), along with rules for bit depth handling, display colors, filename examples for identification, and standard type identifiers. Critical for rule-based prediction, GUI display, and processing stage logic.", "type": "Configuration File"}


category: ProjectGlossary
key: config/llm_settings.json
value:
{"definition": "Configures the experimental LLM-based prediction feature. Contains LLM API connection details, request parameters, the main prompt template, and few-shot examples (input file lists and expected JSON output). Used by LLMEditorWidget and LLM prediction handlers.", "type": "Configuration File"}


category: ProjectGlossary
key: config/suppliers.json
value:
{"definition": "Contains supplier-specific settings, primarily defining the normal map orientation (e.g., OpenGL, DirectX) for known asset suppliers. Used for configuration overrides, GUI auto-completion in supplier selection, and by processing stages like NormalMapGreenChannelStage.", "type": "Configuration File"}


category: ProjectGlossary
key: utils.calculate_sha256
value:
{"definition": "Computes the SHA256 hash of a file. Located in utils/hash_utils.py. Likely used for file integrity checks or unique identification.", "type": "Utility Function"}


category: ProjectGlossary
key: utils.generate_path_from_pattern
value:
{"definition": "Generates output file or directory paths based on a string pattern (e.g., from app_settings.json) and token data. Located in utils/path_utils.py. Crucial for organizing processed assets.", "type": "Utility Function"}


category: ProjectGlossary
key: utils.sanitize_filename
value:
{"definition": "Cleans a string to make it suitable for use as a filename by removing or replacing invalid characters. Located in utils/path_utils.py.", "type": "Utility Function"}


category: ProjectGlossary
key: utils.get_filename_friendly_map_type
value:
{"definition": "Converts an internal map type identifier (e.g., 'MAP_COL' from file_type_definitions.json) into a filename-friendly string (e.g., 'Color'). Located in utils/path_utils.py.", "type": "Utility Function"}


category: ProjectGlossary
key: utils.generate_source_rule_from_archive
value:
{"definition": "Creates an initial SourceRule object by inspecting the contents of an asset archive. Located in utils/prediction_utils.py. Used by the monitor script.", "type": "Utility Function"}


category: ProjectGlossary
key: utils.prepare_processing_workspace
value:
{"definition": "Sets up a temporary workspace for processing, often by extracting an input archive to a temporary location. Located in utils/workspace_utils.py. Used by main processing tasks and the monitor.", "type": "Utility Function"}


category: ProjectGlossary
key: AutoTester class (autotest.py)
value:
{"definition": "The main class in autotest.py that orchestrates the automated end-to-end testing of asset rule prediction and processing by programmatically interacting with the application's MainWindow and comparing results against expected definitions.", "type": "Class", "related_components": ["autotest.py", "Automated End-to-End Rule & Processing Verification (System Pattern ID: 9)"]}


category: ProjectGlossary
key: Expected Rules File (Autotest)
value:
{"definition": "A JSON file defining the anticipated SourceRule structure (including nested AssetRules and FileRules) for a specific test input and preset combination. Used by autotest.py as the ground truth for verifying the application's rule generation logic.", "type": "Data File / Test Artifact", "related_components": ["autotest.py", "Automated End-to-End Rule & Processing Verification (System Pattern ID: 9)"]}


category: ProjectGlossary
key: InfoSummaryFilter (Logging, autotest.py)
value:
{"definition": "A custom Python logging.Filter used in autotest.py to provide concise summary output to the console while allowing detailed logs to be captured elsewhere (e.g., in memory or by the GUI's log console).", "type": "Class (Logging Filter)", "related_components": ["autotest.py"]}


category: ProjectGlossary
key: Preset File Structure
value:
{"definition": "A JSON file defining a specific configuration for asset processing, typically tailored for a particular supplier or asset type. It is merged with app_settings.json by the Configuration class. Key sections include: 'preset_name', 'supplier_name', 'notes', 'source_naming' (rules for parsing filenames including separator, part_indices for base_name/map_type, glossiness_keywords, bit_depth_variants), 'move_to_extra_patterns' (glob patterns for non-primary files), 'map_type_mapping' (keyword-based rules to classify files into standard map types like MAP_COL, MAP_NRM), 'asset_category_rules' (to classify overall asset category e.g., Model, Decal), and 'archetype_rules' (keyword-based rules to assign an archetype like Wood, Metal to an asset).", "type": "Configuration File Structure", "related_components": ["Presets/", "config/app_settings.json", "configuration.py", "gui/preset_editor_widget.py", "gui/prediction_handler.py"]}


category: CriticalPipelineConfiguration
key: INITIAL_SCALING_MODE
value:
{"description": "Determines how initial image scaling is handled in the pipeline.", "typical_values": ["POT_DOWNSCALE", "POT_UPSCALE", "NONE"], "current_value_example": "POT_DOWNSCALE"}


category: CriticalPipelineConfiguration
key: OUTPUT_FORMAT_8BIT
value:
{"description": "Specifies the file format used for 8-bit output textures.", "typical_values": ["png", "jpg"], "current_value_example": "png"}


category: CriticalPipelineConfiguration
key: OUTPUT_BASE_DIR
value:
{"description": "The base directory where all processed assets will be saved.", "current_value_example": "../Asset_Processor_Output_Tests"}


category: ExamplePresetContent
key: PoliigonPresetExample_Simplified
value:
{"preset_name": "Poliigon Standard v2", "supplier_name": "Poliigon", "source_naming": {"separator": "_", "part_indices": {"base_name": 0, "map_type": 1}}, "move_to_extra_patterns": ["*_Preview*", "*.txt"]}


category: CoreSetting
key: TARGET_FILENAME_PATTERN
value:
{"description": "Defines the naming convention for output files.", "default_value": "{base_name}_{map_type}_{resolution}.{ext}"}


category: CoreSetting
key: IMAGE_RESOLUTIONS
value:
{"description": "Defines the standard output resolutions for processed images.", "default_value": {"8K": 8192, "4K": 4096, "2K": 2048, "1K": 1024, "PREVIEW": 128}}


category: CoreSetting
key: MAP_MERGE_RULES
value:
{"description": "Defines how different map types are merged into a single output texture.", "default_value": [{"output_map_type": "MAP_NRMRGH", "inputs": {"R": "MAP_NRM", "G": "MAP_NRM", "B": "MAP_ROUGH"}, "defaults": {"R": 0.5, "G": 0.5, "B": 0.5}, "output_bit_depth": "respect_inputs"}]}


category: CoreSetting
key: OUTPUT_BASE_DIR
value:
{"description": "The base directory where all processed assets will be saved.", "default_value": "../Asset_Processor_Output_Tests"}


category: ConfigurationSchema
key: AssetTypeDefinitionsSchema
value:
{"description": "Defines different asset types supported by the processor. It's a dictionary where keys are asset type identifiers and values are objects with 'color', 'description', and 'examples' properties.", "structure": {"ASSET_TYPE_DEFINITIONS": {"type": "object", "description": "Dictionary of asset type definitions", "properties": {"*": {"type": "object", "description": "Asset type definition", "properties": {"color": {"type": "string", "description": "Hex color code"}, "description": {"type": "string", "description": "Description of the asset type"}, "examples": {"type": "array", "items": {"type": "string"}, "description": "Examples of this asset type"}}, "required": ["color", "description", "examples"]}}, "additionalProperties": true}}, "file_path": "config/asset_type_definitions.json"}


category: ConfigurationSchema
key: FileTypeDefinitionsSchema
value:
{"description": "Defines different file types and their properties used by the processor. It's a dictionary where keys are internal file type identifiers and values are objects with properties like 'bit_depth_rule', 'color', 'description', 'examples', 'is_grayscale', 'keybind', and 'standard_type'.", "structure": {"FILE_TYPE_DEFINITIONS": {"type": "object", "description": "Dictionary of file type definitions", "properties": {"*": {"type": "object", "description": "File type definition", "properties": {"bit_depth_rule": {"type": "string", "description": "Rule for handling bit depth (e.g., 'force_8bit', 'respect')"}, "color": {"type": "string", "description": "Hex color code"}, "description": {"type": "string", "description": "Description of the file type"}, "examples": {"type": "array", "items": {"type": "string"}, "description": "Examples of file name patterns or extensions"}, "is_grayscale": {"type": "boolean", "description": "Whether the file type is typically grayscale"}, "keybind": {"type": "string", "description": "Optional keybind"}, "standard_type": {"type": "string", "description": "Standardized type identifier"}}, "required": ["bit_depth_rule", "color", "description", "examples", "is_grayscale", "keybind", "standard_type"]}}, "additionalProperties": true}}, "file_path": "config/file_type_definitions.json"}


category: ConfigurationSchema
key: SupplierConfigSchema
value:
{"description": "Defines configuration settings per supplier, primarily the normal map type.", "structure": {"type": "object", "description": "Dictionary of supplier configurations", "properties": {"*": {"type": "object", "description": "Supplier configuration", "properties": {"normal_map_type": {"type": "string", "description": "The type of normal map used by this supplier (e.g., 'OpenGL', 'DirectX')"}}, "required": ["normal_map_type"]}}, "additionalProperties": true}, "file_path": "config/suppliers.json"}


category: FutureConsideration
key: ExtendedArchiveSupport
value:
"Future support for other archive formats (e.g., .rar, .7z) in utils/workspace_utils.py would require new utility integrations or library dependencies (e.g., patoolib)."


category: ApplicationDesignPlans
key: FirstTimeSetupPlan_20250515_v2
value:
"## Revised Focused Plan: First-Time Setup for Compiled Asset Processor\n\n**Objective:** To implement a robust and user-friendly first-time setup process that allows users to easily define their configuration directory and initial core settings, ensuring that modifiable configuration files are correctly managed separately from the bundled application, prioritizing portability and preserving existing user configurations.\n\n**Key Process Flow & Components:**\n\n```mermaid\ngraph TD\n    A[Application Start] --> B{User Config Path Saved? (e.g., in AppData/settings.ini)};\n    B -- Yes --> C{Path Valid & Writable?};\n    C -- Yes --> E[Load Configs: Bundled app_settings + User Files from Saved Path];\n    C -- No --> D_INIT[Initiate First-Time Setup Logic];\n    B -- No --> D_INIT;\n\n    D_INIT --> D1{Attempt Portable Default: 'user_data/' next to EXE. Writable?};\n    D1 -- Yes --> F_CONFIRM[Propose Portable Path to User];\n    D1 -- No --> D2{Attempt User Profile Default: e.g., AppData/Roaming/AssetProcessor. Writable?};\n    D2 -- Yes --> F_CONFIRM[Propose User Profile Path to User];\n    D2 -- No --> F_CUSTOM[Force User to Choose Custom Folder];\n\n    F_CONFIRM --> F_UI[Display First-Time Setup UI: Proposed Path + Option to Choose Custom];\n    F_CUSTOM --> F_UI;\n    \n    F_UI --> G_PATH_CHOSEN[User Confirms/Selects Config Location];\n\n    G_PATH_CHOSEN --> H_CHECK_EXISTING{For each default config/preset file...};\n    H_CHECK_EXISTING -- File NOT in User Location --> I_COPY[Copy default file from Bundle to User Location];\n    H_CHECK_EXISTING -- File ALREADY in User Location --> J_SKIP_COPY[Skip copying, keep existing user file];\n    I_COPY --> K_NEXT_FILE[Next file or continue];\n    J_SKIP_COPY --> K_NEXT_FILE;\n    \n    K_NEXT_FILE --> L[User Sets Initial Core Settings (Output Paths, Formats) via UI];\n    L --> M[Save/Update user_settings.json in User Location];\n    M --> N[Save Chosen Config Path (e.g., to AppData/settings.ini)];\n    N --> O[Create '.first_run_complete' marker in User Location (if not already present from existing setup)];\n    O --> E;\n    E --> Z[Application Ready];\n```\n\n**Detailed Steps & Considerations (Revised):**\n\n1.  **First-Run Detection & Config Path Retrieval:**\n    *   On application startup, check for a saved user configuration path (e.g., stored in an OS-specific user application data folder like `AppData/Roaming/AssetProcessor/settings.ini`).\n    *   If found and the path is valid and writable, proceed to load configurations (Step 4).\n    *   If not found, or path is invalid/unwritable, initiate the \"First-Time Setup UI\" sequence.\n\n2.  **Locating Bundled Default Configurations:**\n    *   Use `sys._MEIPASS` or a script-relative fallback. Let's call this `BASE_DIR_APP_BUNDLED`.\n\n3.  **First-Time Setup UI (PySide6 Dialog) & Location Choice:**\n    *   **Default Path Proposal Strategy:**\n        1.  **Attempt Portable Default:** Check if a directory named `user_data` can be created and is writable alongside the main application executable. If yes, this becomes the primary proposed default.\n        2.  **Fallback to User Profile Default:** If the portable location is not feasible, propose a default path in the user's standard application data directory.\n        3.  **Display Proposed Path:** The UI will show the determined proposed default path.\n        4.  **Option for Custom Folder:** Always provide an option for the user to browse and select a different custom folder.\n    *   **Copying Default Files to User Location (Respecting Existing Files):**\n        *   Once the user confirms/selects their configuration location:\n            *   For each definition file (e.g., `asset_type_definitions.json`, etc., **excluding `app_settings.json`**) intended to be copied from `BASE_DIR_APP_BUNDLED/config/`:\n                *   Let the target path be `USER_CHOSEN_PATH/config/filename.json`.\n                *   If `target_path` **does not already exist**, copy the default file from the bundle to `target_path`.\n                *   If `target_path` **already exists**, do nothing.\n            *   Apply the same logic for files and subdirectories within the bundled `Presets/` directory.\n    *   **Initial User Settings (`user_settings.json`):**\n        *   If `USER_CHOSEN_PATH/user_settings.json` already exists, it will be loaded.\n        *   If it does not exist, it's created based on the user's input during the initial setup dialog.\n        *   Required settings: Default Library Output Path, Asset Structure Pattern, Image Output Formats, JPG Threshold.\n    *   **UI Design Consideration:** Design with modularity for future expansion.\n    *   **Completion:**\n        *   Save the chosen configuration path.\n        *   Create/update a marker file (e.g., `.first_run_complete`) in `USER_CHOSEN_PATH`.\n\n4.  **Modified Configuration Loading Logic (within [`configuration.py`](configuration.py:1))**:\n    *   **Loading Sequence:**\n        1.  Load `app_settings.json` *always* from `BASE_DIR_APP_BUNDLED/config/app_settings.json`.\n        2.  Load `user_settings.json` from `BASE_DIR_USER_CONFIG/user_settings.json`.\n        3.  Load other definition files and `Presets/` from `BASE_DIR_USER_CONFIG`.\n    *   **Fallback for Missing User-Modifiable Files (Post First-Time Setup):**\n        *   If a user-modifiable definition file or preset is missing from `BASE_DIR_USER_CONFIG`, attempt to restore it by copying the default from `BASE_DIR_APP_BUNDLED`. Log a warning.\n\n5.  **Modified Configuration Saving Logic (within [`configuration.py`](configuration.py:1))**:\n    *   All saves go to `BASE_DIR_USER_CONFIG`. Bundled defaults are read-only."


category: GUICodeReview_UnifiedViewModel
key: MergingLogic_FileDiscrepancies
value:
{"description": "The merging logic in UnifiedViewModel.update_rules_for_sources() is complex and relies on stable input_path, asset_name, and file_path identifiers. Changes to these identifiers in prediction results (especially from LLM) could lead to mismatches and incorrect merging/removal, potentially causing file list discrepancies.", "file": "gui/unified_view_model.py", "method": "update_rules_for_sources", "lines_reference": "approx 496, 615", "related_issue": "File list discrepancies"}


category: GUICodeReview_UnifiedViewModel
key: CleanupLogic_EmptyAssetRows
value:
{"description": "The cleanup logic in UnifiedViewModel.dropMimeData() (lines 1076-1095) to remove empty parent assets after a move, and in UnifiedViewModel.removeAssetRule() (line 774 check) is complex. If this process fails or incorrectly reports an asset as non-empty, empty asset rows could persist.", "file": "gui/unified_view_model.py", "methods": ["dropMimeData", "removeAssetRule"], "lines_reference": "dropMimeData: approx 1076-1095; removeAssetRule: approx 774", "related_issue": "Persistent empty asset rows"}


category: GUICodeReview_UnifiedViewModel
key: IndexFragility_DragDrop
value:
{"description": "The drag-and-drop implementation in UnifiedViewModel.mimeData() and dropMimeData() stores and reconstructs model indices using simple row numbers. This is vulnerable to model structure changes occurring between drag start and drop end, potentially leading to incorrect item identification and failed moves/updates. QPersistentModelIndex is recommended for robustness.", "file": "gui/unified_view_model.py", "methods": ["mimeData", "dropMimeData"], "lines_reference": "mimeData: approx 908-935; dropMimeData: approx 973-980", "recommendation": "Use QPersistentModelIndex"}


category: GUICodeReview_UnifiedViewModel
key: ParentReference_Maintenance
value:
{"description": "While attempts are made to set parent back-references within UnifiedViewModel, complex sequences of operations could potentially lead to stale references, impacting the reliability of parent() and other methods dependent on accurate parent-child relationships.", "file": "gui/unified_view_model.py", "concern": "Data integrity, model navigation reliability"}


category: GUICodeReview_AssetRestructureHandler
key: EmptyAssetCleanup_ModelDependency
value:
{"description": "AssetRestructureHandler attempts to remove an old parent AssetRule after a FileRule is moved out, but only if old_parent_asset.files is empty. This cleanup is critically dependent on UnifiedViewModel.moveFileRule() correctly and reliably updating the internal file list of the old parent asset. Failures in UnifiedViewModel's list manipulation (e.g., due to index fragility) will prevent the handler's cleanup, leading to persistent empty asset rows.", "file": "gui/asset_restructure_handler.py", "method_context": "handle_target_asset_override", "dependent_module": "gui/unified_view_model.py", "dependent_method": "moveFileRule", "related_issue": "Persistent empty asset rows", "implication": "The root cause of persistent empty rows likely lies in UnifiedViewModel's move/remove logic rather than a complete lack of cleanup attempt in AssetRestructureHandler."}


category: GUICodeReview_AssetRestructureHandler
key: EventDriven_Restructuring
value:
{"description": "AssetRestructureHandler connects to UnifiedViewModel signals (targetAssetOverrideChanged, assetNameChanged) to trigger its restructuring logic (handle_target_asset_override, handle_asset_name_changed). It uses model methods like moveFileRule, createAssetRule, and removeAssetRule.", "file": "gui/asset_restructure_handler.py", "signals_listened": ["targetAssetOverrideChanged", "assetNameChanged"], "model_methods_called": ["moveFileRule", "createAssetRule", "removeAssetRule"]}


category: GUICodeReview_LLMPredictionHandler
key: LLMOutput_SoleFileSource
value:
{"description": "LLMPredictionHandler._parse_llm_response() constructs the SourceRule, AssetRule, and FileRule objects based solely on the file list provided in the LLM's 'individual_file_analysis' output. It does not validate or cross-reference this list against the actual input file list that was originally passed to the handler. This is the direct cause of file list discrepancies in the GUI when the LLM hallucinates or omits files.", "file": "gui/llm_prediction_handler.py", "method_context": "_parse_llm_response", "related_issue": "File list discrepancies", "implication": "To fix discrepancies, the handler (or the UnifiedViewModel when integrating its results) must reconcile the LLM's output with the known actual input files."}


category: GUICodeReview_RuleBasedPredictionHandler
key: ComprehensiveFileProcessing
value:
{"description": "RuleBasedPredictionHandler, via classify_files(), processes and accounts for every file in the provided input list. It builds a SourceRule structure that includes all input files, assigning item types like 'FILE_IGNORE' or 'EXTRA' to those not matching specific rules. This ensures a complete representation of actual input files, contrasting with LLMPredictionHandler which only includes files reported by the LLM. This is why the preset-based approach 'restores' a correct file list in the GUI.", "file": "gui/prediction_handler.py", "method_context": "classify_files, run_prediction", "related_issue": "File list discrepancies (provides a solution/contrast)", "implication": "The strategy of processing all known input files and explicitly categorizing them, rather than relying on an external list that might be incomplete or inaccurate, is key to maintaining file list integrity."}


category: GUICodeReview_UIOrchestration
key: UIOrchestration_IssueLocalization
value:
{"description": "MainWindow and MainPanelWidget correctly instantiate, connect, and orchestrate UnifiedViewModel, AssetRestructureHandler, and prediction handlers. They manage user interactions and data flow triggers. However, the root causes of 'file list discrepancies' and 'persistent empty asset rows' are more likely located within the internal logic of UnifiedViewModel (data merging/updating) and AssetRestructureHandler (restructuring/cleanup) rather than in the UI orchestration layers themselves. The UI components delegate these complex operations appropriately.", "files": ["gui/main_window.py", "gui/main_panel_widget.py"], "related_issues": ["File list discrepancies", "Persistent empty asset rows"], "implication": "Refactoring efforts should primarily target UnifiedViewModel and AssetRestructureHandler, while ensuring UI components correctly adapt to any changes in their APIs or behavior."}


category: GUICodeReview_RuleStructure
key: HierarchyManagement_ExternalDependency
value:
{"description": "The hierarchical relationships in rule_structure.py (SourceRule -> AssetRule -> FileRule) are defined by nested lists (e.g., SourceRule.assets, AssetRule.files). There are no explicit parent back-references (e.g., FileRule.parent_asset) defined as attributes within these data classes themselves. This means the integrity and management of these relationships (including adding, removing, and moving items) are entirely dependent on external components like UnifiedViewModel.", "file": "rule_structure.py", "classes_involved": ["SourceRule", "AssetRule", "FileRule"], "implication": "Potential inconsistencies in the GUI could arise if external management of these nested lists is flawed. The lack of internal back-references might make some model operations more complex or error-prone for the managing classes."}


category: GUICodeReview_RuleStructure
key: Overrides_ClearSeparation
value:
{"description": "The data classes in rule_structure.py clearly separate base predicted/classified attributes (e.g., FileRule.item_type) from user-editable override attributes (e.g., FileRule.item_type_override). This design is beneficial for GUI functionality, allowing user modifications while preserving original predictions.", "file": "rule_structure.py", "classes_involved": ["SourceRule", "AssetRule", "FileRule"], "benefit": "Supports non-destructive user edits and clear data state for the GUI."}


category: GUICodeReview_Phase1_Summary
key: Phase1_RootCauseAnalysis
value:
{"phase_1_status": "Complete", "summary": "Phase 1 (In-Depth Code Review) of the GUI file table refactoring plan is complete. All key modules (UnifiedViewModel, AssetRestructureHandler, LLMPredictionHandler, RuleBasedPredictionHandler, MainWindow, MainPanelWidget, rule_structure.py) have been reviewed, and findings logged to ConPort. Inter-module data flow, state management, and event handling have been synthesized.", "identified_root_causes": {"file_list_discrepancies": "Primarily caused by LLMPredictionHandler generating its SourceRule based only on the LLM's potentially incomplete/inaccurate file list output, and UnifiedViewModel's complex merging logic struggling to reconcile this. RuleBasedPredictionHandler's comprehensive processing of all input files offers a reliable contrast.", "persistent_empty_asset_rows": "Arises from AssetRestructureHandler's cleanup logic being critically dependent on UnifiedViewModel correctly updating its internal list structures (especially parent-child relationships and file counts within assets) after file moves. Failures in UnifiedViewModel's list manipulation, likely exacerbated by index fragility and lack of robust internal parent references in the rule_structure.py definitions, prevent the handler from correctly identifying and removing truly empty parent assets."}, "next_step_planned": "Proceed to Phase 2: Identify Issues & Propose Refactoring Strategies."}


category: GUICodeReview_Phase2_Strategies
key: RefactoringStrategyProposal_v1
value:
{"phase_2_status": "Strategies Proposed", "summary": "Detailed refactoring strategies have been proposed to address file list discrepancies, persistent empty asset rows, and general robustness/maintainability of the GUI file table components.", "strategies": {"file_list_discrepancies": {"description": "Modify LLMPredictionHandler to use the actual input file list as the source of truth. Reconcile LLM classifications by mapping them onto known input files. Flag or ignore LLM outputs for non-existent files. Ensure all actual input files are represented in the SourceRule, possibly as 'UNCLASSIFIED' if LLM provides no info.", "modules_affected": ["gui/llm_prediction_handler.py", "gui/unified_view_model.py"], "benefits": "Ensures GUI reflects actual input files; clear reconciliation with LLM output."}, "persistent_empty_asset_rows": {"description": "Refactor UnifiedViewModel's internal list manipulation for reliability. Implement QPersistentModelIndex for drag-and-drop. Consider adding parent back-references to rule_structure.py (FileRule.parent_asset, AssetRule.parent_source) and update AssetRestructureHandler to use them for more reliable cleanup.", "modules_affected": ["gui/unified_view_model.py", "rule_structure.py", "gui/asset_restructure_handler.py"], "benefits": "Reliable removal of empty assets; robust drag-and-drop; potentially simpler logic."}, "general_robustness_maintainability": {"state_management": "Clarify and document state transitions in UnifiedViewModel; simplify complex logic.", "modularity_coupling": "Reduce direct coupling between GUI components using signals/slots and clearer interfaces.", "logging_error_handling": "Implement more detailed logging in critical sections (ViewModel list manipulation, AssetRestructureHandler cleanup, Prediction Handlers). Improve error handling.", "unit_tests": "Add unit tests for UnifiedViewModel list operations, AssetRestructureHandler cleanup, LLMPredictionHandler reconciliation, and drag-and-drop scenarios.", "modules_affected": ["gui/*", "tests/gui/*"], "benefits": "Improved clarity, debuggability, maintainability, and regression prevention."}}, "next_step_planned": "Review and approve proposed strategies, then proceed to Phase 3: Implementation."}


category: GUIRefactoringDecisions
key: LLMHandler_UnclassifiedFileHandling
value:
{"decision_summary": "During LLMPredictionHandler refactoring, it was decided that input files not classified by the LLM will be assigned a new item type 'FILE_UNCLASSIFIED_BY_LLM' and grouped under a default asset named 'Unclassified Files'.", "module_affected": "gui/llm_prediction_handler.py", "reasoning": "Ensures all actual input files are represented in the SourceRule structure, even if the LLM provides no classification for them, maintaining file list integrity.", "related_strategy": "RefactoringStrategyProposal_v1 (file_list_discrepancies)"}


category: GUIRefactoringImplementations
key: RuleStructure_ParentRefs_Implemented
value:
{"feature_summary": "Added 'parent_asset' attribute to FileRule and 'parent_source' attribute to AssetRule in rule_structure.py. to_json methods updated to exclude these for serialization.", "modules_affected": ["rule_structure.py", "gui/llm_prediction_handler.py"], "notes": "gui/unified_view_model.py and gui/prediction_handler.py (rule-based) were confirmed to correctly manage or adapt to these new references during their operations. This change is foundational for improving hierarchical data management reliability.", "related_strategy": "RefactoringStrategyProposal_v1 (persistent_empty_asset_rows)"}


category: GUIRefactoringImplementations
key: UnifiedViewModel_RobustnessRefactor
value:
{"feature_summary": "Refactored UnifiedViewModel for improved hierarchy management and drag-and-drop. Implemented QPersistentModelIndex in mimeData() and dropMimeData(). Refined cleanup logic in dropMimeData() to reliably use parent back-references. Confirmed other relevant methods correctly leverage parent back-references.", "module_affected": "gui/unified_view_model.py", "benefits": "More stable and reliable drag-and-drop operations. Improved reliability of empty asset cleanup. Enhanced overall robustness of the model.", "related_strategy": "RefactoringStrategyProposal_v1 (persistent_empty_asset_rows and general_robustness_maintainability)"}


category: GUIRefactoringImplementations
key: AssetRestructureHandler_CleanupConfirmed
value:
{"confirmation_summary": "Review of AssetRestructureHandler's handle_target_asset_override method confirmed that its existing logic for cleaning up empty parent AssetRule items is now expected to be robust. This is due to the successful refactoring of UnifiedViewModel (reliable internal list updates, QPersistentModelIndex) and the introduction of parent back-references in rule_structure.py. No code changes were required in AssetRestructureHandler for this specific cleanup mechanism.", "module_affected": "gui/asset_restructure_handler.py", "lines_reference": "approx 135-140", "reasoning": "The handler's dependency on UnifiedViewModel's state (specifically, old_parent_asset.files being empty) is now met more reliably by the refactored model.", "related_strategy": "RefactoringStrategyProposal_v1 (persistent_empty_asset_rows)"}


category: GUIRefactoringImplementations
key: Phase3_CoreRefactors_Summary
value:
{"summary": "Core refactoring tasks for Phase 3 addressing the primary GUI file table issues are complete.", "implemented_changes": [{"issue_addressed": "File List Discrepancies", "module_refactored": "gui/llm_prediction_handler.py", "description": "LLMPredictionHandler now uses the actual input file list as the source of truth, reconciles LLM classifications, handles unclassified files (new 'FILE_UNCLASSIFIED_BY_LLM' type, 'Unclassified Files' asset group), and ignores LLM hallucinations. Logged as GUIRefactoringDecisions:LLMHandler_UnclassifiedFileHandling."}, {"issue_addressed": "Persistent Empty Asset Rows", "modules_refactored": ["rule_structure.py", "gui/unified_view_model.py", "gui/asset_restructure_handler.py"], "description": "Added parent back-references to FileRule and AssetRule in rule_structure.py (logged as GUIRefactoringImplementations:RuleStructure_ParentRefs_Implemented). UnifiedViewModel refactored to use these references, implement QPersistentModelIndex for drag-and-drop, and improve internal list/cleanup logic (logged as GUIRefactoringImplementations:UnifiedViewModel_RobustnessRefactor). AssetRestructureHandler's existing cleanup logic confirmed sufficient with these improvements (logged as GUIRefactoringImplementations:AssetRestructureHandler_CleanupConfirmed)."}], "status": "Core bug-fix refactoring complete. Next steps involve considering general robustness/maintainability improvements (logging, unit tests, etc.) or concluding the refactoring effort.", "related_strategies": "RefactoringStrategyProposal_v1 (file_list_discrepancies, persistent_empty_asset_rows)"}


category: TestAutomation
key: AutotestScriptBehavior
value:
{"summary": "The autotest.py script provides automated headless testing for the Asset Processor GUI workflow. It simulates loading an asset, selecting a preset, predicting rules, processing, and verifying output/logs.", "workflow_steps_overview": "Initialization, Load Expected Rules, Load Asset, Select Preset (triggers prediction), Await Prediction, Compare Rules, Start Processing, Await Processing, Check Output, Analyze Logs, Report Result.", "command_line_options": ["--zipfile (input asset)", "--preset (preset name)", "--expectedrules (JSON file for rule comparison)", "--outputdir (processing output location)", "--search (optional log search term)", "--additional-lines (context lines for search)"], "logging_behavior": {"overall": "Uses a custom logging setup to control console output while capturing all logs in memory.", "console_output": "Filtered by InfoSummaryFilter to show all WARNING/ERRORs and only essential INFOs.", "memory_handler": "Captures all logs (DEBUG and above) for later analysis and searching.", "search_functionality": "The --search argument processes logs from the in-memory buffer, displaying matching lines with specified context."}, "test_files_dependency": "Relies on the TestFiles/ directory for input assets (.zip), expected rules (.json), and default output location (TestFiles/TestOutputs/).", "documentation_reference": "Further details available in Documentation/01_User_Guide/11_Usage_Autotest.md"}


# product_context_history

timestamp: 2025-05-14 20:42:36.160785
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications."}


timestamp: 2025-05-14 20:44:13.500376
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"processing_pipeline": {"description": "A staged processing pipeline responsible for transforming raw asset files into standardized, game-ready textures and metadata. It is orchestrated by the ProcessingEngine and PipelineOrchestrator.", "key_features": ["Modular stage-based processing (pre-item, core item, post-item).", "Contextual data management via AssetProcessingContext.", "Handles individual file processing and merged map creation.", "Supports various transformations (e.g., gloss-to-rough, normal map adjustments, alpha extraction).", "Manages temporary workspaces and output organization."], "core_components": ["ProcessingEngine: Main entry point, initializes orchestrator.", "PipelineOrchestrator: Manages the sequence of stages and the processing loop for assets and their items.", "AssetProcessingContext: Carries data and state through the pipeline for a single asset.", "Processing Stages: Discrete units of work (e.g., SupplierDeterminationStage, PrepareProcessingItemsStage, InitialScalingStage, SaveVariantsStage, OutputOrganizationStage)."]}}}


timestamp: 2025-05-14 20:46:22.640178
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"processing_pipeline": {"description": "A staged processing pipeline responsible for transforming raw asset files into standardized, game-ready textures and metadata. It is orchestrated by the ProcessingEngine and PipelineOrchestrator. Each stage is a modular unit derived from a base `ProcessingStage` class, implementing an `execute(context)` method to perform a specific operation on the `AssetProcessingContext`.", "key_features": ["Modular stage-based processing (pre-item, core item, post-item), with each stage inheriting from `base_stage.ProcessingStage`.", "Contextual data management via AssetProcessingContext.", "Handles individual file processing and merged map creation.", "Supports various transformations (e.g., gloss-to-rough, normal map adjustments, alpha extraction).", "Manages temporary workspaces and output organization."], "core_components": ["ProcessingEngine: Main entry point, initializes orchestrator.", "PipelineOrchestrator: Manages the sequence of stages and the processing loop for assets and their items.", "AssetProcessingContext: Carries data and state through the pipeline for a single asset.", "Processing Stages: Discrete units of work (e.g., `SupplierDeterminationStage`, `AlphaExtractionToMaskStage`, `GlossToRoughConversionStage`, `MetadataFinalizationSaveStage`). Each stage implements the `execute` method from the `ProcessingStage` base."]}}}


timestamp: 2025-05-14 20:48:42.294256
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"prediction_system_rule_generation": {"description": "Responsible for analyzing input files (from archives or directories) and automatically generating an initial hierarchical set of processing rules ([`SourceRule`](rule_structure.py:190) object hierarchy). This hierarchy dictates how each asset and file will be processed by the pipeline. It forms the basis of the 'Hierarchical Rule System' editable in the GUI.", "key_features": ["Supports two main modes of prediction: Rule-Based and LLM-Based.", "Rule-Based: Uses presets ([`gui/prediction_handler.py`](gui/prediction_handler.py:1)) with regex and naming conventions to classify files and determine asset types.", "LLM-Based: Interacts with a Large Language Model ([`gui/llm_prediction_handler.py`](gui/llm_prediction_handler.py:1)) using a structured prompt to interpret files and generate rules.", "Asynchronous operation via `QThread` to keep the GUI responsive, managed by [`BasePredictionHandler`](gui/base_prediction_handler.py:25).", "Provides initial `SourceRule`, `AssetRule`, and `FileRule` structures that can be reviewed and overridden in the GUI."], "core_components": ["[`BasePredictionHandler`](gui/base_prediction_handler.py:25): Abstract base class for prediction, managing threading and signals.", "[`RuleBasedPredictionHandler`](gui/prediction_handler.py:274): Concrete handler for preset-driven rule generation.", "[`LLMPredictionHandler`](gui/llm_prediction_handler.py:22): Concrete handler for LLM-driven rule generation.", "[`classify_files()`](gui/prediction_handler.py:40): Core logic for rule-based file classification using a two-pass regex system."]}}}


timestamp: 2025-05-14 20:50:18.178423
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"configuration_system": {"description": "The Configuration System, centered around the [`Configuration`](configuration.py:93) class in [`configuration.py`](configuration.py:1), is responsible for loading, merging, and providing access to all application settings. This includes core application parameters from [`config/app_settings.json`](config/app_settings.json:1), user-specific overrides from [`config/user_settings.json`](config/user_settings.json:1), supplier-specific processing rules and naming conventions from JSON preset files in `Presets/`, centralized asset type definitions from [`config/asset_type_definitions.json`](config/asset_type_definitions.json:1), file type definitions (including map types, bit-depth rules, and keybinds) from [`config/file_type_definitions.json`](config/file_type_definitions.json:1), known supplier list and settings from [`config/suppliers.json`](config/suppliers.json:1), and LLM integration settings from [`config/llm_settings.json`](config/llm_settings.json:1).", "key_features": ["Hierarchical loading: Core settings -> User overrides -> Preset context.", "Deep merging of dictionaries for flexible overrides.", "Centralized definitions for asset types and file types, enhancing consistency.", "Validation of loaded configurations to ensure integrity.", "Compilation of regex patterns from presets for efficient rule application.", "Provides a consistent, unified view of configuration to other application modules via properties."], "impact_on_flexibility": "This system is crucial for the application's adaptability. By externalizing settings into JSON files and supporting presets, the tool can be easily configured for different asset sources, naming conventions, and processing requirements without code changes. The user override mechanism further allows for personalized adjustments."}}}


timestamp: 2025-05-14 20:53:18.515436
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"configuration_system": {"description": "The Configuration System, centered around the [`Configuration`](configuration.py:93) class in [`configuration.py`](configuration.py:1), is responsible for loading, merging, and providing access to all application settings. This includes core application parameters from [`config/app_settings.json`](config/app_settings.json:1), user-specific overrides from [`config/user_settings.json`](config/user_settings.json:1), supplier-specific processing rules and naming conventions from JSON preset files in `Presets/`, centralized asset type definitions from [`config/asset_type_definitions.json`](config/asset_type_definitions.json:1), file type definitions (including map types, bit-depth rules, and keybinds) from [`config/file_type_definitions.json`](config/file_type_definitions.json:1), known supplier list and settings from [`config/suppliers.json`](config/suppliers.json:1), and LLM integration settings from [`config/llm_settings.json`](config/llm_settings.json:1).", "key_features": ["Hierarchical loading: Core settings -> User overrides -> Preset context.", "Deep merging of dictionaries for flexible overrides.", "Centralized definitions for asset types and file types, enhancing consistency.", "Validation of loaded configurations to ensure integrity.", "Compilation of regex patterns from presets for efficient rule application.", "Provides a consistent, unified view of configuration to other application modules via properties."], "impact_on_flexibility": "This system is crucial for the application's adaptability. By externalizing settings into JSON files and supporting presets, the tool can be easily configured for different asset sources, naming conventions, and processing requirements without code changes. The user override mechanism further allows for personalized adjustments."}}, "hierarchical_rule_system": {"description": "The Asset Processor Tool utilizes a Hierarchical Rule System to provide dynamic and fine-grained control over asset processing. This system allows for overrides of static preset configurations at multiple levels.", "structure": {"SourceRule": "Top-level rule for an entire input source. Defines supplier info, high-level sorting, input path, preset, and contains AssetRules.", "AssetRule": "Mid-level rule for an individual asset. Defines asset name, type, common metadata, and contains FileRules.", "FileRule": "Lowest-level rule for a specific file within an asset. Defines file path, item type, overrides for target asset name/resolution, channel merge instructions, output format, and contains ProcessingItems."}, "purpose": "Enables customized processing instructions, populated by the Prediction System and consumed by the Processing Engine, allowing for flexible adaptation to diverse asset characteristics.", "key_file": "rule_structure.py"}}


timestamp: 2025-05-14 20:54:36.251069
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"graphical_user_interface": {"description": "The Graphical User Interface (GUI), built with PySide6, serves as the primary interactive front-end for the Asset Processor Tool. It allows users to load asset sources (directories or zip files via drag-and-drop or browse), select and manage processing presets (using PresetEditorWidget), configure LLM-based interpretation (LLMEditorWidget), and view/edit the dynamically generated hierarchical rule structure (SourceRule -> AssetRule -> FileRule) in a QTreeView. The MainWindow orchestrates various panels, including the MainPanelWidget which houses the rule view, processing controls, output settings, and Blender integration options. Asynchronous handlers (RuleBasedPredictionHandler, LLMInteractionHandler) ensure UI responsiveness during prediction tasks. A dedicated LogConsoleWidget displays application logs.", "key_features": ["Drag-and-drop loading of asset sources.", "Interactive hierarchical rule view (UnifiedViewModel) for inspecting and overriding predicted processing rules.", "Editing of rule properties (supplier, asset name, asset type, file type, target asset) via custom delegates.", "Drag-and-drop re-parenting of FileRules between AssetRules within the view.", "Context menu for re-interpretation of sources using different presets or LLM.", "Integrated editors for Presets (JSON), LLM settings, Application Preferences, and Core Definitions (Asset/File Types).", "Controls for initiating processing, managing output directory, and Blender post-processing.", "Real-time log display."], "user_interaction_paradigm": "Users load assets, select a preset or LLM for initial rule generation, review/edit the generated rules in the tree view, configure output and Blender options, and then initiate processing. The GUI provides visual feedback on prediction and processing status.", "key_modules": ["gui/main_window.py", "gui/main_panel_widget.py", "gui/unified_view_model.py", "gui/preset_editor_widget.py", "gui/llm_editor_widget.py", "gui/log_console_widget.py", "gui/delegates.py"]}}, "hierarchical_rule_system": {"description": "The Asset Processor Tool utilizes a Hierarchical Rule System to provide dynamic and fine-grained control over asset processing. This system allows for overrides of static preset configurations at multiple levels.", "structure": {"SourceRule": "Top-level rule for an entire input source. Defines supplier info, high-level sorting, input path, preset, and contains AssetRules.", "AssetRule": "Mid-level rule for an individual asset. Defines asset name, type, common metadata, and contains FileRules.", "FileRule": "Lowest-level rule for a specific file within an asset. Defines file path, item type, overrides for target asset name/resolution, channel merge instructions, output format, and contains ProcessingItems."}, "purpose": "Enables customized processing instructions, populated by the Prediction System and consumed by the Processing Engine, allowing for flexible adaptation to diverse asset characteristics.", "key_file": "rule_structure.py"}}


timestamp: 2025-05-14 20:57:45.460302
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"graphical_user_interface": {"description": "The Graphical User Interface (GUI), built with PySide6, serves as the primary interactive front-end for the Asset Processor Tool. It allows users to load asset sources (directories or zip files via drag-and-drop or browse), select and manage processing presets (using PresetEditorWidget), configure LLM-based interpretation (LLMEditorWidget), and view/edit the dynamically generated hierarchical rule structure (SourceRule -> AssetRule -> FileRule) in a QTreeView. The MainWindow orchestrates various panels, including the MainPanelWidget which houses the rule view, processing controls, output settings, and Blender integration options. Asynchronous handlers (RuleBasedPredictionHandler, LLMInteractionHandler) ensure UI responsiveness during prediction tasks. A dedicated LogConsoleWidget displays application logs.", "key_features": ["Drag-and-drop loading of asset sources.", "Interactive hierarchical rule view (UnifiedViewModel) for inspecting and overriding predicted processing rules.", "Editing of rule properties (supplier, asset name, asset type, file type, target asset) via custom delegates.", "Drag-and-drop re-parenting of FileRules between AssetRules within the view.", "Context menu for re-interpretation of sources using different presets or LLM.", "Integrated editors for Presets (JSON), LLM settings, Application Preferences, and Core Definitions (Asset/File Types).", "Controls for initiating processing, managing output directory, and Blender post-processing.", "Real-time log display."], "user_interaction_paradigm": "Users load assets, select a preset or LLM for initial rule generation, review/edit the generated rules in the tree view, configure output and Blender options, and then initiate processing. The GUI provides visual feedback on prediction and processing status.", "key_modules": ["gui/main_window.py", "gui/main_panel_widget.py", "gui/unified_view_model.py", "gui/preset_editor_widget.py", "gui/llm_editor_widget.py", "gui/log_console_widget.py", "gui/delegates.py"]}}, "hierarchical_rule_system": {"description": "The Asset Processor Tool utilizes a Hierarchical Rule System to provide dynamic and fine-grained control over asset processing. This system allows for overrides of static preset configurations at multiple levels.", "structure": {"SourceRule": "Top-level rule for an entire input source. Defines supplier info, high-level sorting, input path, preset, and contains AssetRules.", "AssetRule": "Mid-level rule for an individual asset. Defines asset name, type, common metadata, and contains FileRules.", "FileRule": "Lowest-level rule for a specific file within an asset. Defines file path, item type, overrides for target asset name/resolution, channel merge instructions, output format, and contains ProcessingItems."}, "purpose": "Enables customized processing instructions, populated by the Prediction System and consumed by the Processing Engine, allowing for flexible adaptation to diverse asset characteristics.", "key_file": "rule_structure.py"}, "automated_processing_capabilities": {"directory_monitoring": {"description": "The system includes a Directory Monitor System (implemented in monitor.py) that enables automated, unattended processing of assets. It watches a designated input directory for new archives (zip, rar, 7z).", "rule_generation": "Upon detecting a new archive, the monitor automatically generates processing rules, bypassing the need for manual GUI interaction for these assets.", "engine_invocation": "It then invokes the Processing Engine to process the assets according to the auto-generated rules.", "concurrency": "Uses a ThreadPoolExecutor to handle multiple incoming assets concurrently.", "configuration": "Key aspects like monitored directory, output paths, polling interval, and processing delay are configurable via environment variables."}}}


timestamp: 2025-05-14 20:59:53.776176
change_source: update_product_context
content:
{"initial_product_brief": "# Project Brief: Asset Processor Tool\n\n## 1. Main Goal & Purpose\n\nThe primary goal of the Asset Processor Tool is to provide **CG artists and 3D content teams with a friendly, fast, and flexible interface to process and organize 3D asset source files into a standardized library format.** It automates repetitive and complex tasks involved in preparing assets from various suppliers for use in production pipelines.\n\n## 2. Key Features & Components\n\n*   **Automated Asset Processing:** Ingests 3D asset source files (texture sets, models, etc.) from `.zip`, `.rar`, `.7z` archives, or folders.\n*   **Preset-Driven Workflow:** Utilizes configurable JSON presets to interpret different asset sources (e.g., from various online vendors or internal standards), defining rules for file classification and processing.\n*   **Comprehensive File Operations:**\n    *   **Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, and other file categories based on preset rules.\n    *   **Image Processing:** Performs tasks like image resizing (to standard resolutions like 1K, 2K, 4K, avoiding upscaling), glossiness-to-roughness conversion, normal map green channel inversion (OpenGL/DirectX handling), alpha channel extraction, bit-depth adjustments, and low-resolution fallback generation for small source images.\n    *   **Channel Merging:** Combines channels from different source maps into packed textures (e.g., Normal + Roughness + Metallic into a single NRMRGH map).\n*   **Metadata Generation:** Creates a detailed `metadata.json` file for each processed asset, containing information about maps, categories, processing settings, and more, for downstream tool integration.\n*   **Flexible Output Organization:** Generates a clean, structured output directory based on user-configurable naming patterns and tokens.\n*   **Multiple User Interfaces:**\n    *   **Graphical User Interface (GUI):** The primary interface, designed to be user-friendly, offering drag-and-drop functionality, an integrated preset editor, a live preview table for rule validation and overrides, and clear processing controls.\n    *   **Directory Monitor:** An automated script that watches a specified folder for new asset archives and processes them based on preset names embedded in the archive filename.\n    *   **Command-Line Interface (CLI):** Intended for batch processing and scripting (currently with limited core functionality).\n*   **Optional Blender Integration:** Can automatically run Blender scripts post-processing to create PBR node groups and materials in specified `.blend` files, linking to the newly processed textures.\n*   **Hierarchical Rule System:** Allows for dynamic, granular overrides of preset configurations at the source, asset, or individual file level via the GUI.\n*   **Experimental LLM Prediction:** Includes an option to use a Large Language Model for file interpretation and rule prediction.\n\n## 3. Target Audience\n\n*   **CG Artists:** Individual artists looking for an efficient way to manage and prepare their personal or downloaded asset libraries.\n*   **3D Content Creation Teams:** Studios or groups needing a standardized pipeline for processing and organizing assets from multiple sources.\n*   **Technical Artists/Pipeline Developers:** Who may extend or integrate the tool into broader production workflows.\n\n## 4. Overall Architectural Style & Key Technologies\n\n*   **Core Language:** Python\n*   **GUI Framework:** PySide6\n*   **Configuration:** Primarily JSON-based (application settings, user overrides, type definitions, supplier settings, presets, LLM settings).\n*   **Processing Architecture:** A modular, staged processing pipeline orchestrated by a central engine. Each stage performs a discrete task on an `AssetProcessingContext` object.\n*   **Key Libraries:** OpenCV (image processing), NumPy (numerical operations), py7zr/rarfile (archive handling), watchdog (directory monitoring).\n*   **Design Principles:** Modularity, configurability, and user-friendliness (especially for the GUI).\n\n## 5. Foundational Information\n\n*   The tool aims to significantly reduce manual effort and ensure consistency in asset preparation.\n*   It is designed to be adaptable to various asset sources and pipeline requirements through its extensive configuration options and preset system.\n*   The output `metadata.json` is key for enabling further automation and integration with other tools or digital content creation (DCC) applications.", "system_architecture": {"graphical_user_interface": {"description": "The Graphical User Interface (GUI), built with PySide6, serves as the primary interactive front-end for the Asset Processor Tool. It allows users to load asset sources (directories or zip files via drag-and-drop or browse), select and manage processing presets (using PresetEditorWidget), configure LLM-based interpretation (LLMEditorWidget), and view/edit the dynamically generated hierarchical rule structure (SourceRule -> AssetRule -> FileRule) in a QTreeView. The MainWindow orchestrates various panels, including the MainPanelWidget which houses the rule view, processing controls, output settings, and Blender integration options. Asynchronous handlers (RuleBasedPredictionHandler, LLMInteractionHandler) ensure UI responsiveness during prediction tasks. A dedicated LogConsoleWidget displays application logs.", "key_features": ["Drag-and-drop loading of asset sources.", "Interactive hierarchical rule view (UnifiedViewModel) for inspecting and overriding predicted processing rules.", "Editing of rule properties (supplier, asset name, asset type, file type, target asset) via custom delegates.", "Drag-and-drop re-parenting of FileRules between AssetRules within the view.", "Context menu for re-interpretation of sources using different presets or LLM.", "Integrated editors for Presets (JSON), LLM settings, Application Preferences, and Core Definitions (Asset/File Types).", "Controls for initiating processing, managing output directory, and Blender post-processing.", "Real-time log display."], "user_interaction_paradigm": "Users load assets, select a preset or LLM for initial rule generation, review/edit the generated rules in the tree view, configure output and Blender options, and then initiate processing. The GUI provides visual feedback on prediction and processing status.", "key_modules": ["gui/main_window.py", "gui/main_panel_widget.py", "gui/unified_view_model.py", "gui/preset_editor_widget.py", "gui/llm_editor_widget.py", "gui/log_console_widget.py", "gui/delegates.py"]}}, "hierarchical_rule_system": {"description": "The Asset Processor Tool utilizes a Hierarchical Rule System to provide dynamic and fine-grained control over asset processing. This system allows for overrides of static preset configurations at multiple levels.", "structure": {"SourceRule": "Top-level rule for an entire input source. Defines supplier info, high-level sorting, input path, preset, and contains AssetRules.", "AssetRule": "Mid-level rule for an individual asset. Defines asset name, type, common metadata, and contains FileRules.", "FileRule": "Lowest-level rule for a specific file within an asset. Defines file path, item type, overrides for target asset name/resolution, channel merge instructions, output format, and contains ProcessingItems."}, "purpose": "Enables customized processing instructions, populated by the Prediction System and consumed by the Processing Engine, allowing for flexible adaptation to diverse asset characteristics.", "key_file": "rule_structure.py"}, "automated_processing_capabilities": {"directory_monitoring": {"description": "The system includes a Directory Monitor System (implemented in monitor.py) that enables automated, unattended processing of assets. It watches a designated input directory for new archives (zip, rar, 7z).", "rule_generation": "Upon detecting a new archive, the monitor automatically generates processing rules, bypassing the need for manual GUI interaction for these assets.", "engine_invocation": "It then invokes the Processing Engine to process the assets according to the auto-generated rules.", "concurrency": "Uses a ThreadPoolExecutor to handle multiple incoming assets concurrently.", "configuration": "Key aspects like monitored directory, output paths, polling interval, and processing delay are configurable via environment variables."}}, "blender_integration_system": {"description": "Extends the asset processing pipeline with functionalities executed within Blender. This includes Python scripts for post-processing tasks like automated node group and material creation from processed assets, and a Blender add-on for merging materials.", "key_components": [{"name": "Blender Node Group Generation", "details": "Script ([`blenderscripts/create_nodegroups.py`](blenderscripts/create_nodegroups.py:1)) that creates `PBRSET` (parent) and `PBRTYPE` (child) node groups in a Blender file based on `metadata.json` from processed assets. Textures are loaded and parameters set automatically."}, {"name": "Blender Material Creation", "details": "Script ([`blenderscripts/create_materials.py`](blenderscripts/create_materials.py:1)) that creates Blender materials, linking them to `PBRSET` node groups expected to be in a separate library `.blend` file. Sets previews and viewport properties."}, {"name": "Material Merger Add-on", "details": "Add-on ([`blender_addon/material_merger/`](blender_addon/material_merger/)) providing a UI to merge two materials by combining their node trees using a `MaterialMerge` utility node group."}], "purpose": "To streamline the process of using Asset Processor Tool outputs directly within Blender, automating the setup of materials and node groups, and providing tools for further material customization."}}


# active_context_history

timestamp: 2025-05-16 05:10:30.756594
change_source: update_active_context
content:
{"current_task": "Code review and refactoring plan for GUI file table", "issues_identified": ["File list discrepancies post-LLM processing", "Persistent empty asset rows after renaming/restructuring"], "key_modules_for_review": ["gui/unified_view_model.py", "gui/asset_restructure_handler.py", "gui/llm_prediction_handler.py", "gui/prediction_handler.py", "gui/main_window.py", "gui/main_panel_widget.py", "rule_structure.py"], "proposed_plan_details": "Detailed plan covering code review phases, refactoring strategies for identified issues, and general robustness improvements."}


