{
"sourceFile": "readme.md",
"activeCommit": 2,
"commits": [
{
"activePatchIndex": 0,
"patches": [],
"date": 1745225249658,
"name": "Commit-0",
"content": "# Asset Processor Tool vX.Y\r\n\r\n## Overview\r\n\r\nThis tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.\r\n\r\nThe tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.\r\n\r\nThis tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/`\r\n\r\n## Features\r\n\r\n* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).\r\n* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.\r\n* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).\r\n* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.\r\n * **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `\"COL\"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix.\r\n * **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).\r\n* **Map Processing:**\r\n * Resizes texture maps to configured resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.\r\n * Handles Glossiness map inversion to Roughness.\r\n * Applies bit-depth rules (`respect` source or `force_8bit`).\r\n * Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `\"NRM\"`, `\"DISP\"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution.\r\n* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.\r\n* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, processing settings, etc.\r\n* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).\r\n* **GUI Features:** Drag-and-drop input, integrated preset editor panel, enhanced live preview (shows all files with status: Mapped, Model, Extra, Unrecognised, Ignored, Error), progress bar, cancellation, clear queue button.\r\n* **Responsive GUI:** Utilizes background threads for processing and file preview generation, ensuring the user interface remains responsive.\r\n* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.\r\n* **Docker Support:** Includes a `Dockerfile` for containerized execution.\r\n\r\n## Directory Structure\r\n\r\n```\r\nAsset_processor_tool/\r\n│\r\n├── main.py # CLI Entry Point & processing orchestrator\r\n├── monitor.py # Directory monitoring script for automated processing\r\n├── asset_processor.py # Core class handling single asset processing pipeline\r\n├── configuration.py # Class for loading and accessing configuration\r\n├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)\r\n│\r\n├── gui/ # Contains files related to the Graphical User Interface\r\n│ ├── main_window.py # Main GUI application window and layout\r\n│ ├── processing_handler.py # Handles background processing logic for the GUI\r\n│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI\r\n│\r\n├── Presets/ # Preset definition files\r\n│ ├── _template.json # Template for creating new presets\r\n│ └── Poliigon.json # Example preset for Poliigon assets\r\n│\r\n├── Testfiles/ # Directory containing example input assets for testing\r\n│\r\n├── requirements.txt # Python package dependencies for standard execution\r\n├── requirements-docker.txt # Dependencies specifically for the Docker environment\r\n├── Dockerfile # Instructions for building the Docker container image\r\n└── readme.md # This documentation file\r\n```\r\n\r\n* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`\r\n* **GUI:** `gui/` directory\r\n* **Configuration:** `config.py`, `Presets/` directory\r\n* **Dependencies:** `requirements.txt`, `requirements-docker.txt`\r\n* **Containerization:** `Dockerfile`\r\n* **Documentation/Planning:** `readme.md`, `GUI_PLAN.md`, `MEMORY_OPTIMIZATION_PLAN.md`\r\n* **Testing:** `Testfiles/` directory\r\n\r\n## Architecture\r\n\r\nThis section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.\r\n\r\n### Core Components\r\n\r\nThe tool is primarily built around several key Python modules:\r\n\r\n* **`config.py`**: Defines core, global settings (output paths, resolutions, default behaviors, format rules, etc.) that are generally not supplier-specific.\r\n* **`Presets/*.json`**: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).\r\n* **`configuration.py` (`Configuration` class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.\r\n* **`asset_processor.py` (`AssetProcessor` class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.\r\n* **`main.py`**: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, and calls `AssetProcessor` for each input asset via a wrapper function.\r\n* **`gui/`**: Contains modules related to the Graphical User Interface (GUI), built using PySide6.\r\n* **`monitor.py`**: Implements the directory monitoring functionality for automated processing.\r\n\r\n### Parallel Processing (CLI & GUI)\r\n\r\nTo accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.\r\n\r\n* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.\r\n* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.\r\n* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.\r\n* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).\r\n\r\n### Asset Processing Pipeline (`AssetProcessor` class)\r\n\r\nThe `AssetProcessor` class executes a sequence of steps for each asset:\r\n\r\n1. **`_setup_workspace()`**: Creates a temporary directory for processing.\r\n2. **`_extract_input()`**: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.\r\n3. **`_inventory_and_classify_files()`**: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:\r\n * Identifying files explicitly marked for the `Extra/` folder.\r\n * Identifying model files.\r\n * Matching potential texture maps against keyword patterns.\r\n * Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.\r\n * Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.\r\n * Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).\r\n4. **`_determine_base_metadata()`**: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).\r\n5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.\r\n6. **`_process_maps()`**: Iterates through classified texture maps. For each map:\r\n * Loads the image data (handling potential Gloss->Roughness inversion).\r\n * Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.\r\n * Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).\r\n * Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:\r\n * The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).\r\n * The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).\r\n * The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).\r\n * Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).\r\n * The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).\r\n * Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`).\r\n7. **`_merge_maps()`**: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.\r\n8. **`_generate_metadata_file()`**: Collects all gathered information (asset name, maps present, resolutions, stats, etc.) and writes it to the `metadata.json` file.\r\n9. **`_organize_output_files()`**: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n10. **`_cleanup_workspace()`**: Removes the temporary workspace directory.\r\n\r\n### GUI Architecture (`gui/`)\r\n\r\nThe GUI provides an interactive way to use the tool and manage presets.\r\n\r\n* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.\r\n* **Main Window (`main_window.py`)**: Defines the main application window, which includes:\r\n * An integrated preset editor panel (using `QSplitter`).\r\n * A processing panel with drag-and-drop support, a file preview table, and processing controls.\r\n* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:\r\n * **`ProcessingHandler` (`processing_handler.py`)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) in a background thread.\r\n * **`PredictionHandler` (`prediction_handler.py`)**: Manages the generation of file previews in a background thread. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs the extraction and classification steps without full image processing, making it much faster.\r\n* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors.\r\n* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending.\r\n\r\n### Monitor Architecture (`monitor.py`)\r\n\r\nThe `monitor.py` script enables automated processing of assets dropped into a designated input directory.\r\n\r\n* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.\r\n* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.\r\n* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.\r\n* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.\r\n* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.\r\n* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.\r\n\r\n### Error Handling\r\n\r\n* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.\r\n* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.\r\n* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.\r\n\r\n## Requirements\r\n\r\n* Python 3.8+\r\n* Required Python Packages (see `requirements.txt`):\r\n * `opencv-python` (for image processing)\r\n * `numpy` (for numerical operations)\r\n * `PySide6` (only needed for the GUI)\r\n* Optional Python Packages:\r\n * `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)\r\n\r\nInstall dependencies using pip:\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)\r\n\r\n## Configuration\r\n\r\nThe tool's behavior is controlled by two main configuration components:\r\n\r\n1. **`config.py`:** Defines core, global settings:\r\n * `OUTPUT_BASE_DIR`: Default root directory for processed assets.\r\n * `DEFAULT_ASSET_CATEGORY`: Fallback category (\"Texture\", \"Asset\", \"Decal\").\r\n * `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., \"4K\") to pixel dimensions.\r\n * `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `[\"COL\"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.\r\n * `TARGET_FILENAME_PATTERN`: Format string for output filenames.\r\n * `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).\r\n * `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).\r\n * `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.\r\n * `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `[\"NRM\", \"DISP\"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.\r\n * ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).\r\n\r\n2. **`presets/*.json`:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:\r\n * `supplier_name`: Name of the asset source.\r\n * `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `\"target_type\"` (e.g., `\"COL\"`, `\"NRM\"`) and `\"keywords\"` (a list of source filename patterns like `[\"_col*\", \"_color\"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `\"keywords\"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.\r\n * `bit_depth_variants`: Dictionary mapping standard map types (e.g., `\"NRM\"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `\"*_NRM16*.tif\"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.\r\n * `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).\r\n * `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).\r\n * `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.\r\n * `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.\r\n * `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., \"Decal\").\r\n\r\n Use `presets/_template.json` as a starting point for creating new presets.\r\n\r\n## Usage\r\n\r\n### 1. Graphical User Interface (GUI)\r\n\r\n* **Run:**\r\n ```bash\r\n python gui/main_window.py\r\n ```\r\n* **Interface:**\r\n * **Preset Editor Panel (Left):** Allows creating, deleting, loading, editing, and saving presets directly within the main window. Select a preset from the list to load it into the editor tabs.\r\n * **Preset Selector (Top Right):** Select the preset to use for *processing* the current queue.\r\n * **Drag and Drop:** Drag asset ZIP files or folders onto the designated area in the right panel.\r\n * **Preview:** The table shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.\r\n * `Mapped`: Files recognised as standard texture maps (e.g., Color, Normal).\r\n * `Model`: Files recognised as 3D models (e.g., FBX, OBJ).\r\n * `Extra`: Files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation).\r\n * `Unrecognised`: Files that did not match any map, model, or explicit extra pattern.\r\n * `Ignored`: Files intentionally ignored (e.g., superseded by a 16-bit variant).\r\n * `Error`: Indicates an issue during prediction for this file or asset.\r\n * **Options:**\r\n * `Overwrite Existing`: Check to force reprocessing if output already exists.\r\n * `Workers`: Set the number of assets to process concurrently.\r\n * **Controls:**\r\n * `Clear Queue`: Removes all assets from the current processing queue and clears the preview.\r\n * `Start Processing`: Begins the processing pipeline for all added assets.\r\n * `Cancel`: Attempts to stop ongoing processing.\r\n * **Progress Bar:** Shows the overall processing progress.\r\n * **Status Bar:** Displays messages about the current state, errors, or completion.\r\n\r\n### 2. Command-Line Interface (CLI)\r\n\r\n* **Run:**\r\n ```bash\r\n python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]\r\n ```\r\n* **Arguments:**\r\n * `INPUT_PATH`: One or more paths to input ZIP files or folders.\r\n * `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).\r\n * `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.\r\n * `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).\r\n * `--overwrite`: Force reprocessing and overwrite existing output.\r\n * `-v`, `--verbose`: Enable detailed DEBUG level logging.\r\n* **Example:**\r\n ```bash\r\n python main.py \"C:/Downloads/WoodFine001.zip\" \"C:/Downloads/MetalScratched02.zip\" -p Poliigon -o \"G:/Assets/Processed\" --workers 4 --overwrite\r\n ```\r\n\r\n### 3. Directory Monitor (Automated Processing)\r\n\r\n* **Run:**\r\n ```bash\r\n python monitor.py\r\n ```\r\n* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name.\r\n* **Configuration (Environment Variables):**\r\n * `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).\r\n * `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).\r\n * `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).\r\n * `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).\r\n * `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).\r\n * `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).\r\n * `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).\r\n * `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).\r\n* **Output:**\r\n * Logs processing activity to the console.\r\n * Processed assets are created in the `OUTPUT_DIR` following the standard structure.\r\n * The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.\r\n\r\n## Processing Pipeline (Simplified)\r\n\r\n1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.\r\n2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\r\n3. **Metadata Determination:** Asset name, category, and archetype are determined.\r\n4. **Skip Check:** If output exists and overwrite is off, processing stops here.\r\n5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\r\n6. **Merging:** Channels are merged according to preset rules and saved.\r\n7. **Metadata Generation:** `metadata.json` is created with all collected information.\r\n8. **Output Organization:** Processed files are moved to the final structured output directory.\r\n9. **Cleanup:** The temporary workspace is removed.\r\n\r\n## Output Structure\r\n\r\nProcessed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\r\n\r\nEach asset directory typically contains:\r\n* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\r\n* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\r\n* Model files (if present in source).\r\n* `metadata.json`: Detailed information about the asset and processing.\r\n* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\r\n\r\n## Docker\r\n\r\nA `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands."
},
{
"activePatchIndex": 0,
"patches": [],
"date": 1745228181337,
"name": "Commit1-21-04- 11:36",
"content": "# Asset Processor Tool vX.Y\r\n\r\n## Overview\r\n\r\nThis tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.\r\n\r\nThe tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.\r\n\r\nThis tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/` for reference\r\n\r\n## Features\r\n\r\n* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).\r\n* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.\r\n* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).\r\n* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.\r\n * **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `\"COL\"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix.\r\n * **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).\r\n* **Map Processing:**\r\n * Resizes texture maps to configured resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.\r\n * Handles Glossiness map inversion to Roughness.\r\n * Applies bit-depth rules (`respect` source or `force_8bit`).\r\n * Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `\"NRM\"`, `\"DISP\"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution.\r\n * Calculates and stores the relative aspect ratio change string in metadata.\r\n* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.\r\n* **Aspect Ratio Metadata:** Calculates the relative aspect ratio change during resizing and stores it in the `metadata.json` file (`aspect_ratio_change_string`). The format indicates if the aspect is unchanged (`EVEN`), scaled horizontally (`X110`, `X122`, etc.) or scaled vertically (`Y133`, `Y112`, etc.)\r\n* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, aspect ratio change, processing settings, etc.\r\n* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).\r\n* **GUI Features:** Drag-and-drop input, integrated preset editor panel, enhanced live preview (shows all files with status: Mapped, Model, Extra, Unrecognised, Ignored, Error), progress bar, cancellation, clear queue button.\r\n* **Responsive GUI:** Utilizes background threads for processing and file preview generation, ensuring the user interface remains responsive.\r\n* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.\r\n* **Docker Support:** Includes a `Dockerfile` for containerized execution.\r\n\r\n## Directory Structure\r\n\r\n```\r\nAsset_processor_tool/\r\n│\r\n├── main.py # CLI Entry Point & processing orchestrator\r\n├── monitor.py # Directory monitoring script for automated processing\r\n├── asset_processor.py # Core class handling single asset processing pipeline\r\n├── configuration.py # Class for loading and accessing configuration\r\n├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)\r\n│\r\n├── gui/ # Contains files related to the Graphical User Interface\r\n│ ├── main_window.py # Main GUI application window and layout\r\n│ ├── processing_handler.py # Handles background processing logic for the GUI\r\n│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI\r\n│\r\n├── Presets/ # Preset definition files\r\n│ ├── _template.json # Template for creating new presets\r\n│ └── Poliigon.json # Example preset for Poliigon assets\r\n│\r\n├── Testfiles/ # Directory containing example input assets for testing\r\n│\r\n├── requirements.txt # Python package dependencies for standard execution\r\n├── requirements-docker.txt # Dependencies specifically for the Docker environment\r\n├── Dockerfile # Instructions for building the Docker container image\r\n└── readme.md # This documentation file\r\n```\r\n\r\n* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`\r\n* **GUI:** `gui/` directory\r\n* **Configuration:** `config.py`, `Presets/` directory\r\n* **Dependencies:** `requirements.txt`, `requirements-docker.txt`\r\n* **Containerization:** `Dockerfile`\r\n* **Documentation/Planning:** `readme.md`, `GUI_PLAN.md`, `MEMORY_OPTIMIZATION_PLAN.md`\r\n* **Testing:** `Testfiles/` directory\r\n\r\n## Architecture\r\n\r\nThis section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.\r\n\r\n### Core Components\r\n\r\nThe tool is primarily built around several key Python modules:\r\n\r\n* **`config.py`**: Defines core, global settings (output paths, resolutions, default behaviors, format rules, etc.) that are generally not supplier-specific.\r\n* **`Presets/*.json`**: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).\r\n* **`configuration.py` (`Configuration` class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.\r\n* **`asset_processor.py` (`AssetProcessor` class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.\r\n* **`main.py`**: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, and calls `AssetProcessor` for each input asset via a wrapper function.\r\n* **`gui/`**: Contains modules related to the Graphical User Interface (GUI), built using PySide6.\r\n* **`monitor.py`**: Implements the directory monitoring functionality for automated processing.\r\n\r\n### Parallel Processing (CLI & GUI)\r\n\r\nTo accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.\r\n\r\n* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.\r\n* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.\r\n* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.\r\n* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).\r\n\r\n### Asset Processing Pipeline (`AssetProcessor` class)\r\n\r\nThe `AssetProcessor` class executes a sequence of steps for each asset:\r\n\r\n1. **`_setup_workspace()`**: Creates a temporary directory for processing.\r\n2. **`_extract_input()`**: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.\r\n3. **`_inventory_and_classify_files()`**: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:\r\n * Identifying files explicitly marked for the `Extra/` folder.\r\n * Identifying model files.\r\n * Matching potential texture maps against keyword patterns.\r\n * Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.\r\n * Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.\r\n * Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).\r\n4. **`_determine_base_metadata()`**: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).\r\n5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.\r\n6. **`_process_maps()`**: Iterates through classified texture maps. For each map:\r\n * Loads the image data (handling potential Gloss->Roughness inversion).\r\n * Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.\r\n * Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).\r\n * Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:\r\n * The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).\r\n * The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).\r\n * The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).\r\n * Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).\r\n * The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).\r\n * Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`) and determines the aspect ratio change string (e.g., \"EVEN\", \"X150\", \"Y075\") stored in the metadata.\r\n7. **`_merge_maps()`**: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.\r\n8. **`_generate_metadata_file()`**: Collects all gathered information (asset name, maps present, resolutions, stats, aspect ratio change, etc.) and writes it to the `metadata.json` file.\r\n9. **`_organize_output_files()`**: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n10. **`_cleanup_workspace()`**: Removes the temporary workspace directory.\r\n\r\n### GUI Architecture (`gui/`)\r\n\r\nThe GUI provides an interactive way to use the tool and manage presets.\r\n\r\n* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.\r\n* **Main Window (`main_window.py`)**: Defines the main application window, which includes:\r\n * An integrated preset editor panel (using `QSplitter`).\r\n * A processing panel with drag-and-drop support, a file preview table, and processing controls.\r\n* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:\r\n * **`ProcessingHandler` (`processing_handler.py`)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) in a background thread.\r\n * **`PredictionHandler` (`prediction_handler.py`)**: Manages the generation of file previews in a background thread. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs the extraction and classification steps without full image processing, making it much faster.\r\n* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors.\r\n* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending.\r\n\r\n### Monitor Architecture (`monitor.py`)\r\n\r\nThe `monitor.py` script enables automated processing of assets dropped into a designated input directory.\r\n\r\n* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.\r\n* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.\r\n* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.\r\n* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.\r\n* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.\r\n* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.\r\n\r\n### Error Handling\r\n\r\n* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.\r\n* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.\r\n* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.\r\n\r\n## Requirements\r\n\r\n* Python 3.8+\r\n* Required Python Packages (see `requirements.txt`):\r\n * `opencv-python` (for image processing)\r\n * `numpy` (for numerical operations)\r\n * `PySide6` (only needed for the GUI)\r\n* Optional Python Packages:\r\n * `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)\r\n\r\nInstall dependencies using pip:\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)\r\n\r\n## Configuration\r\n\r\nThe tool's behavior is controlled by two main configuration components:\r\n\r\n1. **`config.py`:** Defines core, global settings:\r\n * `OUTPUT_BASE_DIR`: Default root directory for processed assets.\r\n * `DEFAULT_ASSET_CATEGORY`: Fallback category (\"Texture\", \"Asset\", \"Decal\").\r\n * `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., \"4K\") to pixel dimensions.\r\n * `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `[\"COL\"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.\r\n * `TARGET_FILENAME_PATTERN`: Format string for output filenames.\r\n * `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).\r\n * `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).\r\n * `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.\r\n * `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `[\"NRM\", \"DISP\"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.\r\n * ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).\r\n\r\n2. **`presets/*.json`:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:\r\n * `supplier_name`: Name of the asset source.\r\n * `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `\"target_type\"` (e.g., `\"COL\"`, `\"NRM\"`) and `\"keywords\"` (a list of source filename patterns like `[\"_col*\", \"_color\"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `\"keywords\"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.\r\n * `bit_depth_variants`: Dictionary mapping standard map types (e.g., `\"NRM\"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `\"*_NRM16*.tif\"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.\r\n * `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).\r\n * `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).\r\n * `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.\r\n * `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.\r\n * `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., \"Decal\").\r\n\r\n Use `presets/_template.json` as a starting point for creating new presets.\r\n\r\n## Usage\r\n\r\n### 1. Graphical User Interface (GUI)\r\n\r\n* **Run:**\r\n ```bash\r\n python gui/main_window.py\r\n ```\r\n* **Interface:**\r\n * **Preset Editor Panel (Left):** Allows creating, deleting, loading, editing, and saving presets directly within the main window. Select a preset from the list to load it into the editor tabs.\r\n * **Preset Selector (Top Right):** Select the preset to use for *processing* the current queue.\r\n * **Drag and Drop:** Drag asset ZIP files or folders onto the designated area in the right panel.\r\n * **Preview:** The table shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.\r\n * `Mapped`: Files recognised as standard texture maps (e.g., Color, Normal).\r\n * `Model`: Files recognised as 3D models (e.g., FBX, OBJ).\r\n * `Extra`: Files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation).\r\n * `Unrecognised`: Files that did not match any map, model, or explicit extra pattern.\r\n * `Ignored`: Files intentionally ignored (e.g., superseded by a 16-bit variant).\r\n * `Error`: Indicates an issue during prediction for this file or asset.\r\n * **Options:**\r\n * `Overwrite Existing`: Check to force reprocessing if output already exists.\r\n * `Workers`: Set the number of assets to process concurrently.\r\n * **Controls:**\r\n * `Clear Queue`: Removes all assets from the current processing queue and clears the preview.\r\n * `Start Processing`: Begins the processing pipeline for all added assets.\r\n * `Cancel`: Attempts to stop ongoing processing.\r\n * **Progress Bar:** Shows the overall processing progress.\r\n * **Status Bar:** Displays messages about the current state, errors, or completion.\r\n\r\n### 2. Command-Line Interface (CLI)\r\n\r\n* **Run:**\r\n ```bash\r\n python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]\r\n ```\r\n* **Arguments:**\r\n * `INPUT_PATH`: One or more paths to input ZIP files or folders.\r\n * `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).\r\n * `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.\r\n * `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).\r\n * `--overwrite`: Force reprocessing and overwrite existing output.\r\n * `-v`, `--verbose`: Enable detailed DEBUG level logging.\r\n* **Example:**\r\n ```bash\r\n python main.py \"C:/Downloads/WoodFine001.zip\" \"C:/Downloads/MetalScratched02.zip\" -p Poliigon -o \"G:/Assets/Processed\" --workers 4 --overwrite\r\n ```\r\n\r\n### 3. Directory Monitor (Automated Processing)\r\n\r\n* **Run:**\r\n ```bash\r\n python monitor.py\r\n ```\r\n* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name.\r\n* **Configuration (Environment Variables):**\r\n * `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).\r\n * `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).\r\n * `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).\r\n * `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).\r\n * `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).\r\n * `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).\r\n * `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).\r\n * `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).\r\n* **Output:**\r\n * Logs processing activity to the console.\r\n * Processed assets are created in the `OUTPUT_DIR` following the standard structure.\r\n * The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.\r\n\r\n## Processing Pipeline (Simplified)\r\n\r\n1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.\r\n2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\r\n3. **Metadata Determination:** Asset name, category, and archetype are determined.\r\n4. **Skip Check:** If output exists and overwrite is off, processing stops here.\r\n5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\r\n6. **Merging:** Channels are merged according to preset rules and saved.\r\n7. **Metadata Generation:** `metadata.json` is created with all collected information.\r\n8. **Output Organization:** Processed files are moved to the final structured output directory.\r\n9. **Cleanup:** The temporary workspace is removed.\r\n\r\n## Output Structure\r\n\r\nProcessed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\r\n\r\nEach asset directory typically contains:\r\n* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\r\n* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\r\n* Model files (if present in source).\r\n* `metadata.json`: Detailed information about the asset and processing.\r\n* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\r\n\r\n## Docker\r\n\r\nA `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands."
},
{
"activePatchIndex": 3,
"patches": [
{
"date": 1745235041942,
"content": "Index: \n===================================================================\n--- \n+++ \n"
},
{
"date": 1745235051623,
"content": "Index: \n===================================================================\n--- \n+++ \n@@ -0,0 +1,329 @@\n+# Asset Processor Tool vX.Y\r\n+\r\n+## Overview\r\n+\r\n+This tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.\r\n+\r\n+The tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.\r\n+\r\n+This tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/` for reference\r\n+\r\n+## Features\r\n+\r\n+* **Aspect Ratio Metadata:** Calculates the relative aspect ratio change during resizing and stores it in the `metadata.json` file (`aspect_ratio_change_string`). The format indicates if the aspect is unchanged (`EVEN`), scaled horizontally (`X150`, `X075`, etc.), scaled vertically (`Y150`, `Y075`, etc.), or scaled non-uniformly (`X150Y120`).\r\n+* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).\r\n+* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.\r\n+* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).\r\n+* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.\r\n+ * **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `\"COL\"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix.\r\n+ * **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).\r\n+* **Map Processing:**\r\n+ * Resizes texture maps to configured resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.\r\n+ * Handles Glossiness map inversion to Roughness.\r\n+ * Applies bit-depth rules (`respect` source or `force_8bit`).\r\n+ * Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `\"NRM\"`, `\"DISP\"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).\r\n+ * Calculates basic image statistics (Min/Max/Mean) for a reference resolution.\r\n+ * Calculates and stores the relative aspect ratio change string in metadata.\r\n+* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.\r\n+* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, aspect ratio change, processing settings, etc.\r\n+* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n+* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).\r\n+* **GUI Features:** Drag-and-drop input, integrated preset editor panel, enhanced live preview (shows all files with status: Mapped, Model, Extra, Unrecognised, Ignored, Error), progress bar, cancellation, clear queue button.\r\n+* **Responsive GUI:** Utilizes background threads for processing and file preview generation, ensuring the user interface remains responsive.\r\n+* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.\r\n+* **Docker Support:** Includes a `Dockerfile` for containerized execution.\r\n+\r\n+## Directory Structure\r\n+\r\n+```\r\n+Asset_processor_tool/\r\n+│\r\n+├── main.py # CLI Entry Point & processing orchestrator\r\n+├── monitor.py # Directory monitoring script for automated processing\r\n+├── asset_processor.py # Core class handling single asset processing pipeline\r\n+├── configuration.py # Class for loading and accessing configuration\r\n+├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)\r\n+│\r\n+├── blenderscripts/ # Scripts for integration with Blender\r\n+│ └── create_nodegroups.py # Script to create node groups from processed assets\r\n+│\r\n+├── gui/ # Contains files related to the Graphical User Interface\r\n+│ ├── main_window.py # Main GUI application window and layout\r\n+│ ├── processing_handler.py # Handles background processing logic for the GUI\r\n+│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI\r\n+│\r\n+├── Presets/ # Preset definition files\r\n+│ ├── _template.json # Template for creating new presets\r\n+│ └── Poliigon.json # Example preset for Poliigon assets\r\n+│\r\n+├── Testfiles/ # Directory containing example input assets for testing\r\n+│\r\n+├── requirements.txt # Python package dependencies for standard execution\r\n+├── requirements-docker.txt # Dependencies specifically for the Docker environment\r\n+├── Dockerfile # Instructions for building the Docker container image\r\n+└── readme.md # This documentation file\r\n+```\r\n+\r\n+* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`\r\n+* **Blender Integration:** `blenderscripts/` directory\r\n+* **GUI:** `gui/` directory\r\n+* **Configuration:** `config.py`, `Presets/` directory\r\n+* **Dependencies:** `requirements.txt`, `requirements-docker.txt`\r\n+* **Containerization:** `Dockerfile`\r\n+* **Documentation/Planning:** `readme.md`, `GUI_PLAN.md`, `MEMORY_OPTIMIZATION_PLAN.md`\r\n+* **Testing:** `Testfiles/` directory\r\n+\r\n+## Architecture\r\n+\r\n+This section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.\r\n+\r\n+### Core Components\r\n+\r\n+The tool is primarily built around several key Python modules:\r\n+\r\n+* **`config.py`**: Defines core, global settings (output paths, resolutions, default behaviors, format rules, etc.) that are generally not supplier-specific.\r\n+* **`Presets/*.json`**: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).\r\n+* **`configuration.py` (`Configuration` class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.\r\n+* **`asset_processor.py` (`AssetProcessor` class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.\r\n+* **`main.py`**: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, and calls `AssetProcessor` for each input asset via a wrapper function.\r\n+* **`gui/`**: Contains modules related to the Graphical User Interface (GUI), built using PySide6.\r\n+* **`monitor.py`**: Implements the directory monitoring functionality for automated processing.\r\n+\r\n+### Parallel Processing (CLI & GUI)\r\n+\r\n+To accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.\r\n+\r\n+* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.\r\n+* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.\r\n+* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.\r\n+* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).\r\n+\r\n+### Asset Processing Pipeline (`AssetProcessor` class)\r\n+\r\n+The `AssetProcessor` class executes a sequence of steps for each asset:\r\n+\r\n+1. **`_setup_workspace()`**: Creates a temporary directory for processing.\r\n+2. **`_extract_input()`**: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.\r\n+3. **`_inventory_and_classify_files()`**: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:\r\n+ * Identifying files explicitly marked for the `Extra/` folder.\r\n+ * Identifying model files.\r\n+ * Matching potential texture maps against keyword patterns.\r\n+ * Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.\r\n+ * Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.\r\n+ * Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).\r\n+4. **`_determine_base_metadata()`**: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).\r\n+5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.\r\n+6. **`_process_maps()`**: Iterates through classified texture maps. For each map:\r\n+ * Loads the image data (handling potential Gloss->Roughness inversion).\r\n+ * Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.\r\n+ * Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).\r\n+ * Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:\r\n+ * The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).\r\n+ * The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).\r\n+ * The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).\r\n+ * Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).\r\n+ * The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).\r\n+ * Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).\r\n+ * Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`) and determines the aspect ratio change string (e.g., \"EVEN\", \"X150\", \"Y075\") stored in the metadata.\r\n+7. **`_merge_maps()`**: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.\r\n+8. **`_generate_metadata_file()`**: Collects all gathered information (asset name, maps present, resolutions, stats, aspect ratio change, etc.) and writes it to the `metadata.json` file.\r\n+9. **`_organize_output_files()`**: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n+10. **`_cleanup_workspace()`**: Removes the temporary workspace directory.\r\n+\r\n+### GUI Architecture (`gui/`)\r\n+\r\n+The GUI provides an interactive way to use the tool and manage presets.\r\n+\r\n+* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.\r\n+* **Main Window (`main_window.py`)**: Defines the main application window, which includes:\r\n+ * An integrated preset editor panel (using `QSplitter`).\r\n+ * A processing panel with drag-and-drop support, a file preview table, and processing controls.\r\n+* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:\r\n+ * **`ProcessingHandler` (`processing_handler.py`)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) in a background thread.\r\n+ * **`PredictionHandler` (`prediction_handler.py`)**: Manages the generation of file previews in a background thread. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs the extraction and classification steps without full image processing, making it much faster.\r\n+* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors.\r\n+* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending.\r\n+\r\n+### Monitor Architecture (`monitor.py`)\r\n+\r\n+The `monitor.py` script enables automated processing of assets dropped into a designated input directory.\r\n+\r\n+* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.\r\n+* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.\r\n+* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.\r\n+* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.\r\n+* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.\r\n+* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.\r\n+\r\n+### Error Handling\r\n+\r\n+* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.\r\n+* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.\r\n+* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.\r\n+\r\n+## Requirements\r\n+\r\n+* Python 3.8+\r\n+* Required Python Packages (see `requirements.txt`):\r\n+ * `opencv-python` (for image processing)\r\n+ * `numpy` (for numerical operations)\r\n+ * `PySide6` (only needed for the GUI)\r\n+* Optional Python Packages:\r\n+ * `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)\r\n+\r\n+Install dependencies using pip:\r\n+```bash\r\n+pip install -r requirements.txt\r\n+```\r\n+(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)\r\n+\r\n+## Configuration\r\n+\r\n+The tool's behavior is controlled by two main configuration components:\r\n+\r\n+1. **`config.py`:** Defines core, global settings:\r\n+ * `OUTPUT_BASE_DIR`: Default root directory for processed assets.\r\n+ * `DEFAULT_ASSET_CATEGORY`: Fallback category (\"Texture\", \"Asset\", \"Decal\").\r\n+ * `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., \"4K\") to pixel dimensions.\r\n+ * `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `[\"COL\"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.\r\n+ * `TARGET_FILENAME_PATTERN`: Format string for output filenames.\r\n+ * `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).\r\n+ * `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).\r\n+ * `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.\r\n+ * `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `[\"NRM\", \"DISP\"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.\r\n+ * ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).\r\n+\r\n+2. **`presets/*.json`:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:\r\n+ * `supplier_name`: Name of the asset source.\r\n+ * `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `\"target_type\"` (e.g., `\"COL\"`, `\"NRM\"`) and `\"keywords\"` (a list of source filename patterns like `[\"_col*\", \"_color\"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `\"keywords\"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.\r\n+ * `bit_depth_variants`: Dictionary mapping standard map types (e.g., `\"NRM\"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `\"*_NRM16*.tif\"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.\r\n+ * `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).\r\n+ * `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).\r\n+ * `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.\r\n+ * `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.\r\n+ * `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., \"Decal\").\r\n+\r\n+ Use `presets/_template.json` as a starting point for creating new presets.\r\n+\r\n+## Usage\r\n+\r\n+### 1. Graphical User Interface (GUI)\r\n+\r\n+* **Run:**\r\n+ ```bash\r\n+ python gui/main_window.py\r\n+ ```\r\n+* **Interface:**\r\n+ * **Preset Editor Panel (Left):** Allows creating, deleting, loading, editing, and saving presets directly within the main window. Select a preset from the list to load it into the editor tabs.\r\n+ * **Preset Selector (Top Right):** Select the preset to use for *processing* the current queue.\r\n+ * **Drag and Drop:** Drag asset ZIP files or folders onto the designated area in the right panel.\r\n+ * **Preview:** The table shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.\r\n+ * `Mapped`: Files recognised as standard texture maps (e.g., Color, Normal).\r\n+ * `Model`: Files recognised as 3D models (e.g., FBX, OBJ).\r\n+ * `Extra`: Files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation).\r\n+ * `Unrecognised`: Files that did not match any map, model, or explicit extra pattern.\r\n+ * `Ignored`: Files intentionally ignored (e.g., superseded by a 16-bit variant).\r\n+ * `Error`: Indicates an issue during prediction for this file or asset.\r\n+ * **Options:**\r\n+ * `Overwrite Existing`: Check to force reprocessing if output already exists.\r\n+ * `Workers`: Set the number of assets to process concurrently.\r\n+ * **Controls:**\r\n+ * `Clear Queue`: Removes all assets from the current processing queue and clears the preview.\r\n+ * `Start Processing`: Begins the processing pipeline for all added assets.\r\n+ * `Cancel`: Attempts to stop ongoing processing.\r\n+ * **Progress Bar:** Shows the overall processing progress.\r\n+ * **Status Bar:** Displays messages about the current state, errors, or completion.\r\n+\r\n+### 2. Command-Line Interface (CLI)\r\n+\r\n+* **Run:**\r\n+ ```bash\r\n+ python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]\r\n+ ```\r\n+* **Arguments:**\r\n+ * `INPUT_PATH`: One or more paths to input ZIP files or folders.\r\n+ * `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).\r\n+ * `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.\r\n+ * `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).\r\n+ * `--overwrite`: Force reprocessing and overwrite existing output.\r\n+ * `-v`, `--verbose`: Enable detailed DEBUG level logging.\r\n+* **Example:**\r\n+ ```bash\r\n+ python main.py \"C:/Downloads/WoodFine001.zip\" \"C:/Downloads/MetalScratched02.zip\" -p Poliigon -o \"G:/Assets/Processed\" --workers 4 --overwrite\r\n+ ```\r\n+\r\n+### 3. Directory Monitor (Automated Processing)\r\n+\r\n+* **Run:**\r\n+ ```bash\r\n+ python monitor.py\r\n+ ```\r\n+* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name.\r\n+* **Configuration (Environment Variables):**\r\n+ * `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).\r\n+ * `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).\r\n+ * `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).\r\n+ * `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).\r\n+ * `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).\r\n+ * `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).\r\n+ * `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).\r\n+ * `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).\r\n+* **Output:**\r\n+ * Logs processing activity to the console.\r\n+ * Processed assets are created in the `OUTPUT_DIR` following the standard structure.\r\n+ * The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.\r\n+ \r\n+ ### 4. Blender Node Group Creation (Manual Script)\r\n+ \r\n+ * **Purpose:** After processing assets with this tool, you can use the provided Blender script to automatically create and configure PBR node groups within a Blender file, leveraging the generated metadata.\r\n+ * **Script:** `blenderscripts/create_nodegroups.py`\r\n+ * **Prerequisites:**\r\n+ * A library of assets processed by this tool, located at a known path.\r\n+ * A Blender file containing two template node groups named exactly `Template_PBRSET` and `Template_PBRTYPE`. These templates should contain the necessary input/output sockets and internal nodes (e.g., Image Texture nodes labeled by resolution like \"1K\", \"2K\", \"4K\"; Value nodes labeled \"AspectRatioCorrection\" and \"HighestResolution\"; CombineXYZ nodes labeled \"Histogram-[MAPTYPE]\").\r\n+ * **Configuration (Inside the script):**\r\n+ * `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located.\r\n+ * Other variables like template names, node labels, color space mappings, etc., can be adjusted if needed.\r\n+ * **Run:**\r\n+ 1. Open your target Blender file (containing the templates).\r\n+ 2. Open the `blenderscripts/create_nodegroups.py` script in Blender's Text Editor.\r\n+ 3. Ensure `PROCESSED_ASSET_LIBRARY_ROOT` is correctly set in the script's configuration section.\r\n+ 4. Click \"Run Script\" (or Alt+P). Check the Blender System Console (Window -> Toggle System Console) for progress and potential errors.\r\n+ * **Functionality:**\r\n+ * Scans the configured library path for processed assets (identified by `metadata.json`).\r\n+ * For each asset, it creates/updates a parent node group (e.g., `PBRSET_AssetName`).\r\n+ * For each map type (including merged maps like NRMRGH), it creates/updates a child node group using a Base64 encoded name (e.g., `PBRTYPE_[EncodedAssetName_MapType]`) and links it to the corresponding placeholder in the parent group.\r\n+ * Loads the processed texture images (handling mixed extensions via fallback) into the correct Image Texture nodes within the child groups and sets their color spaces.\r\n+ * Applies pre-calculated metadata to the parent group:\r\n+ * Sets the `AspectRatioCorrection` node value based on image dimensions and the `aspect_ratio_change_string`.\r\n+ * Sets `Histogram-[MAPTYPE]` node values based on `image_stats_1k` from metadata.\r\n+ * Sets the `HighestResolution` node value (1K=1.0, 2K=2.0, 4K=3.0, 8K=4.0).\r\n+ * Adds `supplier_name` and `archetype` as asset tags.\r\n+ * Sets the custom asset preview using the lowest resolution color map.\r\n+ * Uses an optional manifest file (`[ActiveBlendFileName]_manifest.json`) stored alongside the `.blend` file to track progress and skip already processed maps/resolutions on subsequent runs (requires the `.blend` file to be saved).\r\n+ \r\n+ ## Processing Pipeline (Simplified)\r\n+ \r\n+1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.\r\n+2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\r\n+3. **Metadata Determination:** Asset name, category, and archetype are determined.\r\n+4. **Skip Check:** If output exists and overwrite is off, processing stops here.\r\n+5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\r\n+6. **Merging:** Channels are merged according to preset rules and saved.\r\n+7. **Metadata Generation:** `metadata.json` is created with all collected information.\r\n+8. **Output Organization:** Processed files are moved to the final structured output directory.\r\n+9. **Cleanup:** The temporary workspace is removed.\r\n+\r\n+## Output Structure\r\n+\r\n+Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\r\n+\r\n+Each asset directory typically contains:\r\n+* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\r\n+* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\r\n+* Model files (if present in source).\r\n+* `metadata.json`: Detailed information about the asset and processing.\r\n+* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\r\n+\r\n+## Docker\r\n+\r\n+A `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands.\n\\ No newline at end of file\n"
},
{
"date": 1745237329911,
"content": "Index: \n===================================================================\n--- \n+++ \n@@ -315,5 +315,23 @@\n 2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\r\n 3. **Metadata Determination:** Asset name, category, and archetype are determined.\r\n 4. **Skip Check:** If output exists and overwrite is off, processing stops here.\r\n 5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\r\n-6. **Merging:** Channels are m\n\\ No newline at end of file\n+6. **Merging:** Channels are merged according to preset rules and saved.\r\n+7. **Metadata Generation:** `metadata.json` is created with all collected information.\r\n+8. **Output Organization:** Processed files are moved to the final structured output directory.\r\n+9. **Cleanup:** The temporary workspace is removed.\r\n+\r\n+## Output Structure\r\n+\r\n+Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\r\n+\r\n+Each asset directory typically contains:\r\n+* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\r\n+* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\r\n+* Model files (if present in source).\r\n+* `metadata.json`: Detailed information about the asset and processing.\r\n+* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\r\n+\r\n+## Docker\r\n+\r\n+A `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands.\n\\ No newline at end of file\n"
},
{
"date": 1745319515003,
"content": "Index: \n===================================================================\n--- \n+++ \n@@ -0,0 +1,355 @@\n+# Asset Processor Tool vX.Y\n+\n+## Overview\n+\n+This tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.\n+\n+The tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.\n+\n+This tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/` for reference\n+\n+## Features\n+\n+* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).\n+* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.\n+* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).\n+* **Multi-Asset Input Handling:** Correctly identifies and processes multiple distinct assets contained within a single input ZIP or folder, creating separate outputs for each.\n+* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.\n+ * **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `\"COL\"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix. **Note:** There is a known issue (see #ISSUE-006) where variant numbering may not reset correctly for each asset in a multi-asset input.\n+ * **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).\n+* **Map Processing:**\n+ * Resizes texture maps to configured power of two resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.\n+ * Handles Glossiness map inversion to Roughness.\n+ * Applies bit-depth rules (`respect` source or `force_8bit`).\n+ * Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `\"NRM\"`, `\"DISP\"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).\n+ * Calculates basic image statistics (Min/Max/Mean) for a reference resolution.\n+ * Calculates and stores the relative aspect ratio change string in metadata.\n+* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.\n+* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, aspect ratio change, processing settings, etc. **Aspect Ratio Metadata:** Calculates the relative aspect ratio change during resizing and stores it in the `metadata.json` file (`aspect_ratio_change_string`). The format indicates if the aspect is unchanged (`EVEN`), scaled horizontally (`X150`, `X110`, etc.), scaled vertically (`Y150`, `Y125`, etc.)\n+* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).\n+* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).\n+* **Blender Integration:** Optionally runs Blender scripts (`create_nodegroups.py`, `create_materials.py`) after asset processing to automate node group and material creation in specified `.blend` files. Available via both CLI and GUI.\n+* **GUI Features:**\n+ * Drag-and-drop input for assets (ZIPs/folders).\n+ * Integrated preset editor panel for managing `.json` presets.\n+ * Configurable output directory field with a browse button (defaults to path in `config.py`).\n+ * Enhanced live preview table showing predicted file status (Mapped, Model, Extra, Unrecognised, Ignored, Error) based on the selected processing preset.\n+ * Toggleable preview mode (via View menu) to switch between detailed file preview and a simple list of input assets.\n+ * Toggleable log console panel (via View menu) displaying application log messages within the GUI.\n+ * Progress bar, cancellation button, and clear queue button.\n+ * **Blender Post-Processing Controls:** Checkbox to enable/disable Blender script execution and input fields with browse buttons to specify the target `.blend` files for node group and material creation (defaults configurable in `config.py`).\n+* **Responsive GUI:** Utilizes background threads (`QThread`) for processing (`ProcessPoolExecutor`) and file preview generation (`ThreadPoolExecutor`), ensuring the user interface remains responsive during intensive operations.\n+* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.\n+* **Docker Support:** Includes a `Dockerfile` for containerized execution.\n+\n+## Directory Structure\n+\n+```\n+Asset_processor_tool/\n+│\n+├── main.py # CLI Entry Point & processing orchestrator\n+├── monitor.py # Directory monitoring script for automated processing\n+├── asset_processor.py # Core class handling single asset processing pipeline\n+├── configuration.py # Class for loading and accessing configuration\n+├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)\n+│\n+├── blenderscripts/ # Scripts for integration with Blender\n+│ └── create_nodegroups.py # Script to create node groups from processed assets\n+│ └── create_materials.py # Script to create materials linking to node groups\n+│\n+├── gui/ # Contains files related to the Graphical User Interface\n+│ ├── main_window.py # Main GUI application window and layout\n+│ ├── processing_handler.py # Handles background processing logic for the GUI\n+│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI\n+│\n+├── Presets/ # Preset definition files\n+│ ├── _template.json # Template for creating new presets\n+│ └── Poliigon.json # Example preset for Poliigon assets\n+│\n+├── Testfiles/ # Directory containing example input assets for testing\n+│\n+├── Tickets/ # Directory for issue and feature tracking (Markdown files)\n+│ ├── _template.md # Template for creating new tickets\n+│ └── Ticket-README.md # Explanation of the ticketing system\n+│\n+├── requirements.txt # Python package dependencies for standard execution\n+├── requirements-docker.txt # Dependencies specifically for the Docker environment\n+├── Dockerfile # Instructions for building the Docker container image\n+└── readme.md # This documentation file\n+```\n+\n+* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`\n+* **Blender Integration:** `blenderscripts/` directory\n+* **GUI:** `gui/` directory\n+* **Configuration:** `config.py`, `Presets/` directory\n+* **Dependencies:** `requirements.txt`, `requirements-docker.txt`\n+* **Containerization:** `Dockerfile`\n+* **Documentation/Planning:** `readme.md`, `Project Notes/` directory\n+* **Issue/Feature Tracking:** `Tickets/` directory (see `Tickets/README.md`)\n+* **Testing:** `Testfiles/` directory\n+\n+## Architecture\n+\n+This section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.\n+\n+### Core Components\n+\n+The tool is primarily built around several key Python modules:\n+\n+* `config.py`: Defines core, global settings (output paths, resolutions, default behaviors, format rules, Blender executable path, default Blender file paths, etc.) that are generally not supplier-specific.\n+* `Presets/*.json`: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).\n+* `configuration.py` **(**`Configuration` **class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.\n+* `asset_processor.py` **(**`AssetProcessor` **class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.\n+* `main.py`: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, calls `AssetProcessor` for each input asset via a wrapper function, and optionally triggers Blender script execution after processing.\n+* `gui/`: Contains modules related to the Graphical User Interface (GUI), built using PySide6.\n+* `monitor.py`: Implements the directory monitoring functionality for automated processing.\n+\n+### Parallel Processing (CLI & GUI)\n+\n+To accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.\n+\n+* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.\n+* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.\n+* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.\n+* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).\n+\n+### Asset Processing Pipeline (`AssetProcessor` class)\n+\n+The `AssetProcessor` class executes a sequence of steps for each asset:\n+\n+ 1. `_setup_workspace()`: Creates a temporary directory for processing.\n+ 2. `_extract_input()`: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.\n+ 3. `_inventory_and_classify_files()`: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:\n+ * Identifying files explicitly marked for the `Extra/` folder.\n+ * Identifying model files.\n+ * Matching potential texture maps against keyword patterns.\n+ * Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.\n+ * Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.\n+ * Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).\n+ 4. `_determine_base_metadata()`: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).\n+ 5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.\n+ 6. `_process_maps()`: Iterates through classified texture maps. For each map:\n+ * Loads the image data (handling potential Gloss->Roughness inversion).\n+ * Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.\n+ * Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).\n+ * Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors: \n+ * The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).\n+ * The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).\n+ * The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).\n+ * Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).\n+ * The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).\n+ * Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).\n+ * Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`) and determines the aspect ratio change string (e.g., \"EVEN\", \"X150\", \"Y075\") stored in the metadata.\n+ 7. `_merge_maps()`: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.\n+ 8. `_generate_metadata_file()`: Collects all gathered information (asset name, maps present, resolutions, stats, aspect ratio change, etc.) and writes it to the `metadata.json` file.\n+ 9. `_organize_output_files()`: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).\n+10. `_cleanup_workspace()`: Removes the temporary workspace directory.\n+\n+### GUI Architecture (`gui/`)\n+\n+The GUI provides an interactive way to use the tool and manage presets.\n+\n+* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.\n+* **Main Window (**`main_window.py`**)**: Defines the main application window, which includes: \n+ * An integrated preset editor panel (using `QSplitter`).\n+ * A processing panel with drag-and-drop support, output directory selection, a file preview table, and processing controls.\n+ * **Blender Post-Processing Controls:** A group box containing a checkbox to enable/disable Blender script execution and input fields with browse buttons for specifying the target `.blend` files for node group and material creation.\n+* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s: \n+ * `ProcessingHandler` **(**`processing_handler.py`**)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) and the optional Blender script execution in a background thread. Receives the target output directory and Blender integration settings from the main window.\n+ * `PredictionHandler` **(**`prediction_handler.py`**)**: Manages the generation of file previews in a background thread using a `ThreadPoolExecutor` to parallelize prediction across multiple assets. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs extraction and classification.\n+* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors. A custom `QtLogHandler` redirects Python log messages to the UI console via signals.\n+* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending. Includes an optional, toggleable log console panel at the top.\n+\n+### Monitor Architecture (`monitor.py`)\n+\n+The `monitor.py` script enables automated processing of assets dropped into a designated input directory.\n+\n+* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.\n+* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.\n+* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.\n+* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.\n+* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.\n+* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.\n+\n+### Error Handling\n+\n+* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.\n+* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.\n+* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.\n+\n+## Requirements\n+\n+* Python 3.8+\n+* Required Python Packages (see `requirements.txt`): \n+ * `opencv-python` (for image processing)\n+ * `numpy` (for numerical operations)\n+ * `PySide6` (only needed for the GUI)\n+* Optional Python Packages: \n+ * `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)\n+* **Blender:** A working installation of Blender is required for the optional Blender integration features. The path to the executable should be configured in `config.py` or available in the system's PATH.\n+\n+Install dependencies using pip:\n+\n+```bash\n+pip install -r requirements.txt\n+```\n+\n+(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)\n+\n+## Configuration\n+\n+The tool's behavior is controlled by two main configuration components:\n+\n+1. `config.py`**:** Defines core, global settings:\n+ * `OUTPUT_BASE_DIR`: Default root directory for processed assets.\n+ * `DEFAULT_ASSET_CATEGORY`: Fallback category (\"Texture\", \"Asset\", \"Decal\").\n+ * `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., \"4K\") to pixel dimensions.\n+ * `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `[\"COL\"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.\n+ * `TARGET_FILENAME_PATTERN`: Format string for output filenames.\n+ * `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).\n+ * `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).\n+ * `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.\n+ * `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `[\"NRM\", \"DISP\"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.\n+ * `BLENDER_EXECUTABLE_PATH`: Path to the Blender executable (required for Blender integration).\n+ * `DEFAULT_NODEGROUP_BLEND_PATH`: Default path to the .blend file for node group creation (used by GUI if not specified).\n+ * `DEFAULT_MATERIALS_BLEND_PATH`: Default path to the .blend file for material creation (used by GUI if not specified).\n+ * ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).\n+2. `presets/*.json`**:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:\n+ * `supplier_name`: Name of the asset source.\n+ * `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `\"target_type\"` (e.g., `\"COL\"`, `\"NRM\"`) and `\"keywords\"` (a list of source filename patterns like `[\"_col*\", \"_color\"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `\"keywords\"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.\n+ * `bit_depth_variants`: Dictionary mapping standard map types (e.g., `\"NRM\"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `\"*_NRM16*.tif\"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.\n+ * `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).\n+ * `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).\n+ * `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.\n+ * `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.\n+ * `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., \"Decal\").\n+\n+ Use `presets/_template.json` as a starting point for creating new presets.\n+\n+## Usage\n+\n+### 1. Graphical User Interface (GUI)\n+\n+* **Run:**\n+\n+ ```bash\n+ python gui/main_window.py\n+ ```\n+* **Interface:**\n+ * **Menu Bar:** Contains a \"View\" menu to toggle visibility of the Log Console and enable/disable the detailed file preview.\n+ * **Preset Editor Panel (Left):**\n+ * Optional **Log Console:** A text area at the top displaying application log messages (toggle via View menu).\n+ * **Preset List:** Allows creating, deleting, loading, editing, and saving presets. Select a preset here to load it into the editor tabs below.\n+ * **Preset Editor Tabs:** Edit preset details (\"General & Naming\", \"Mapping & Rules\").\n+ * **Processing Panel (Right):**\n+ * **Preset Selector:** Select the preset to use for *processing* the current queue.\n+ * **Output Directory:** Displays the target output directory. Defaults to the path in `config.py`. Use the \"Browse...\" button to select a different directory.\n+ * **Drag and Drop Area:** Drag asset ZIP files or folders here to add them to the queue.\n+ * **Preview Table:** Displays information about the assets in the queue. Behavior depends on the \"Disable Detailed Preview\" option in the View menu: \n+ * **Detailed Preview (Default):** Shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.\n+ * **Simple View (Preview Disabled):** Shows only the list of top-level input asset paths (ZIPs/folders) added to the queue.\n+ * **Progress Bar:** Shows the overall processing progress.\n+ * **Blender Post-Processing:** A group box containing a checkbox to enable/disable the optional Blender script execution. When enabled, input fields and browse buttons appear to specify the `.blend` files for node group and material creation. These fields default to the paths configured in `config.py`.\n+ * **Options & Controls (Bottom):**\n+ * `Overwrite Existing`: Checkbox to force reprocessing if output already exists.\n+ * `Workers`: Spinbox to set the number of assets to process concurrently.\n+ * `Clear Queue`: Button to remove all assets from the queue and clear the preview.\n+ * `Start Processing`: Button to begin processing all assets in the queue.\n+ * `Cancel`: Button to attempt stopping ongoing processing.\n+ * **Status Bar:** Displays messages about the current state, errors, or completion.\n+\n+### 2. Command-Line Interface (CLI)\n+\n+* **Run:**\n+\n+ ```bash\n+ python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]\n+ ```\n+* **Arguments:**\n+ * `INPUT_PATH`: One or more paths to input ZIP files or folders.\n+ * `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).\n+ * `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.\n+ * `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).\n+ * `--overwrite`: Force reprocessing and overwrite existing output.\n+ * `-v`, `--verbose`: Enable detailed DEBUG level logging.\n+ * `--nodegroup-blend NODEGROUP_BLEND`: Path to the .blend file for creating/updating node groups. Overrides `config.py` default. If provided, triggers node group script execution after processing.\n+ * `--materials-blend MATERIALS_BLEND`: Path to the .blend file for creating/updating materials. Overrides `config.py` default. If provided, triggers material script execution after processing.\n+* **Example:**\n+\n+ ```bash\n+ python main.py \"C:/Downloads/WoodFine001.zip\" -p Poliigon -o \"G:/Assets/Processed\" --workers 4 --overwrite --nodegroup-blend \"G:/Blender/Libraries/NodeGroups.blend\" --materials-blend \"G:/Blender/Libraries/Materials.blend\"\n+ ```\n+\n+### 3. Directory Monitor (Automated Processing)\n+\n+* **Run:**\n+\n+ ```bash\n+ python monitor.py\n+ ```\n+* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name. **Note:** The directory monitor currently does *not* support the optional Blender script execution. This feature is only available via the CLI and GUI.\n+* **Configuration (Environment Variables):**\n+ * `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).\n+ * `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).\n+ * `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).\n+ * `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).\n+ * `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).\n+ * `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).\n+ * `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).\n+ * `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).\n+* **Output:**\n+ * Logs processing activity to the console.\n+ * Processed assets are created in the `OUTPUT_DIR` following the standard structure.\n+ * The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.\n+\n+ ### 4. Blender Node Group Creation Script (`blenderscripts/create_nodegroups.py`)\n+ * **Purpose:** This script, designed to be run *within* Blender (either manually or triggered by `main.py`/GUI), scans processed assets and creates/updates PBR node groups in the active `.blend` file.\n+ * **Execution:** Typically run via the Asset Processor tool's CLI or GUI after asset processing. Can also be run manually in Blender's Text Editor.\n+ * **Prerequisites (for manual run):**\n+ * A library of assets processed by this tool, located at a known path.\n+ * A Blender file containing two template node groups named exactly `Template_PBRSET` and `Template_PBRTYPE`.\n+ * **Configuration (Inside the script for manual run):**\n+ * `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located. This is overridden by the tool when run via CLI/GUI.\n+ * **Functionality:** Reads metadata, creates/updates node groups, loads textures, sets up nodes, applies metadata-driven settings (aspect ratio, stats, highest resolution), and sets asset previews. Includes an explicit save command at the end.\n+\n+ ### 5. Blender Material Creation Script (`blenderscripts/create_materials.py`)\n+ * **Purpose:** This script, designed to be run *within* Blender (either manually or triggered by `main.py`/GUI), scans processed assets and creates/updates materials in the active `.blend` file that link to the PBRSET node groups created by `create_nodegroups.py`.\n+ * **Execution:** Typically run via the Asset Processor tool's CLI or GUI after asset processing. Can also be run manually in Blender's Text Editor.\n+ * **Prerequisites (for manual run):**\n+ * A library of assets processed by this tool, located at a known path.\n+ * A `.blend` file containing the PBRSET node groups created by `create_nodegroups.py`.\n+ * A template material in the *current* Blender file named `Template_PBRMaterial` that uses nodes and contains a Group node labeled `PLACEHOLDER_NODE_LABEL`.\n+ * **Configuration (Inside the script for manual run):**\n+ * `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located. This is overridden by the tool when run via CLI/GUI.\n+ * `NODEGROUP_BLEND_FILE_PATH`: **Must be updated** within the script to point to the `.blend` file containing the PBRSET node groups. This is overridden by the tool when run via CLI/GUI.\n+ * `TEMPLATE_MATERIAL_NAME`, `PLACEHOLDER_NODE_LABEL`, `MATERIAL_NAME_PREFIX`, `PBRSET_GROUP_PREFIX`, etc., can be adjusted if needed.\n+ * **Functionality:** Reads metadata, creates/updates materials by copying the template, links the corresponding PBRSET node group from the specified `.blend` file, marks materials as assets, copies tags, sets custom previews, and sets viewport properties based on metadata. Includes an explicit save command at the end.\n+\n+## Processing Pipeline (Simplified)\n+\n+ 1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.\n+ 2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\n+ 3. **Metadata Determination:** Asset name, category, and archetype are determined.\n+ 4. **Skip Check:** If output exists and overwrite is off, processing stops here.\n+ 5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\n+ 6. **Merging:** Channels are merged according to preset rules and saved.\n+ 7. **Metadata Generation:** `metadata.json` is created with all collected information.\n+ 8. **Output Organization:** Processed files are moved to the final structured output directory.\n+ 9. **Cleanup:** The temporary workspace is removed.\n+10. **Optional Blender Script Execution:** If configured via CLI or GUI, Blender is launched in the background to run `create_nodegroups.py` and `create_materials.py` on specified `.blend` files, using the processed asset output directory as input.\n+\n+## Output Structure\n+\n+Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\n+\n+Each asset directory typically contains:\n+\n+* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\n+* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\n+* Model files (if present in source).\n+* `metadata.json`: Detailed information about the asset and processing.\n+* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\n+\n+## Docker\n+\n+A `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands.\n\\ No newline at end of file\n"
}
],
"date": 1745235041942,
"name": "Commit2-21-04- 13:30",
"content": "# Asset Processor Tool vX.Y\r\n\r\n## Overview\r\n\r\nThis tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.\r\n\r\nThe tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.\r\n\r\nThis tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/` for reference\r\n\r\n## Features\r\n\r\n* **Aspect Ratio Metadata:** Calculates the relative aspect ratio change during resizing and stores it in the `metadata.json` file (`aspect_ratio_change_string`). The format indicates if the aspect is unchanged (`EVEN`), scaled horizontally (`X150`, `X075`, etc.), scaled vertically (`Y150`, `Y075`, etc.), or scaled non-uniformly (`X150Y120`).\r\n* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).\r\n* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.\r\n* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).\r\n* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.\r\n * **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `\"COL\"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix.\r\n * **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).\r\n* **Map Processing:**\r\n * Resizes texture maps to configured resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.\r\n * Handles Glossiness map inversion to Roughness.\r\n * Applies bit-depth rules (`respect` source or `force_8bit`).\r\n * Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `\"NRM\"`, `\"DISP\"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution.\r\n * Calculates and stores the relative aspect ratio change string in metadata.\r\n* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.\r\n* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, aspect ratio change, processing settings, etc.\r\n* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).\r\n* **GUI Features:** Drag-and-drop input, integrated preset editor panel, enhanced live preview (shows all files with status: Mapped, Model, Extra, Unrecognised, Ignored, Error), progress bar, cancellation, clear queue button.\r\n* **Responsive GUI:** Utilizes background threads for processing and file preview generation, ensuring the user interface remains responsive.\r\n* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.\r\n* **Docker Support:** Includes a `Dockerfile` for containerized execution.\r\n\r\n## Directory Structure\r\n\r\n```\r\nAsset_processor_tool/\r\n│\r\n├── main.py # CLI Entry Point & processing orchestrator\r\n├── monitor.py # Directory monitoring script for automated processing\r\n├── asset_processor.py # Core class handling single asset processing pipeline\r\n├── configuration.py # Class for loading and accessing configuration\r\n├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)\r\n│\r\n├── blenderscripts/ # Scripts for integration with Blender\r\n│ └── create_nodegroups.py # Script to create node groups from processed assets\r\n│\r\n├── gui/ # Contains files related to the Graphical User Interface\r\n│ ├── main_window.py # Main GUI application window and layout\r\n│ ├── processing_handler.py # Handles background processing logic for the GUI\r\n│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI\r\n│\r\n├── Presets/ # Preset definition files\r\n│ ├── _template.json # Template for creating new presets\r\n│ └── Poliigon.json # Example preset for Poliigon assets\r\n│\r\n├── Testfiles/ # Directory containing example input assets for testing\r\n│\r\n├── requirements.txt # Python package dependencies for standard execution\r\n├── requirements-docker.txt # Dependencies specifically for the Docker environment\r\n├── Dockerfile # Instructions for building the Docker container image\r\n└── readme.md # This documentation file\r\n```\r\n\r\n* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`\r\n* **Blender Integration:** `blenderscripts/` directory\r\n* **GUI:** `gui/` directory\r\n* **Configuration:** `config.py`, `Presets/` directory\r\n* **Dependencies:** `requirements.txt`, `requirements-docker.txt`\r\n* **Containerization:** `Dockerfile`\r\n* **Documentation/Planning:** `readme.md`, `GUI_PLAN.md`, `MEMORY_OPTIMIZATION_PLAN.md`\r\n* **Testing:** `Testfiles/` directory\r\n\r\n## Architecture\r\n\r\nThis section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.\r\n\r\n### Core Components\r\n\r\nThe tool is primarily built around several key Python modules:\r\n\r\n* **`config.py`**: Defines core, global settings (output paths, resolutions, default behaviors, format rules, etc.) that are generally not supplier-specific.\r\n* **`Presets/*.json`**: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).\r\n* **`configuration.py` (`Configuration` class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.\r\n* **`asset_processor.py` (`AssetProcessor` class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.\r\n* **`main.py`**: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, and calls `AssetProcessor` for each input asset via a wrapper function.\r\n* **`gui/`**: Contains modules related to the Graphical User Interface (GUI), built using PySide6.\r\n* **`monitor.py`**: Implements the directory monitoring functionality for automated processing.\r\n\r\n### Parallel Processing (CLI & GUI)\r\n\r\nTo accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.\r\n\r\n* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.\r\n* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.\r\n* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.\r\n* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).\r\n\r\n### Asset Processing Pipeline (`AssetProcessor` class)\r\n\r\nThe `AssetProcessor` class executes a sequence of steps for each asset:\r\n\r\n1. **`_setup_workspace()`**: Creates a temporary directory for processing.\r\n2. **`_extract_input()`**: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.\r\n3. **`_inventory_and_classify_files()`**: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:\r\n * Identifying files explicitly marked for the `Extra/` folder.\r\n * Identifying model files.\r\n * Matching potential texture maps against keyword patterns.\r\n * Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.\r\n * Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.\r\n * Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).\r\n4. **`_determine_base_metadata()`**: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).\r\n5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.\r\n6. **`_process_maps()`**: Iterates through classified texture maps. For each map:\r\n * Loads the image data (handling potential Gloss->Roughness inversion).\r\n * Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.\r\n * Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).\r\n * Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:\r\n * The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).\r\n * The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).\r\n * The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).\r\n * Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).\r\n * The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).\r\n * Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).\r\n * Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`) and determines the aspect ratio change string (e.g., \"EVEN\", \"X150\", \"Y075\") stored in the metadata.\r\n7. **`_merge_maps()`**: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.\r\n8. **`_generate_metadata_file()`**: Collects all gathered information (asset name, maps present, resolutions, stats, aspect ratio change, etc.) and writes it to the `metadata.json` file.\r\n9. **`_organize_output_files()`**: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).\r\n10. **`_cleanup_workspace()`**: Removes the temporary workspace directory.\r\n\r\n### GUI Architecture (`gui/`)\r\n\r\nThe GUI provides an interactive way to use the tool and manage presets.\r\n\r\n* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.\r\n* **Main Window (`main_window.py`)**: Defines the main application window, which includes:\r\n * An integrated preset editor panel (using `QSplitter`).\r\n * A processing panel with drag-and-drop support, a file preview table, and processing controls.\r\n* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:\r\n * **`ProcessingHandler` (`processing_handler.py`)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) in a background thread.\r\n * **`PredictionHandler` (`prediction_handler.py`)**: Manages the generation of file previews in a background thread. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs the extraction and classification steps without full image processing, making it much faster.\r\n* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors.\r\n* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending.\r\n\r\n### Monitor Architecture (`monitor.py`)\r\n\r\nThe `monitor.py` script enables automated processing of assets dropped into a designated input directory.\r\n\r\n* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.\r\n* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.\r\n* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.\r\n* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.\r\n* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.\r\n* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.\r\n\r\n### Error Handling\r\n\r\n* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.\r\n* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.\r\n* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.\r\n\r\n## Requirements\r\n\r\n* Python 3.8+\r\n* Required Python Packages (see `requirements.txt`):\r\n * `opencv-python` (for image processing)\r\n * `numpy` (for numerical operations)\r\n * `PySide6` (only needed for the GUI)\r\n* Optional Python Packages:\r\n * `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)\r\n\r\nInstall dependencies using pip:\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)\r\n\r\n## Configuration\r\n\r\nThe tool's behavior is controlled by two main configuration components:\r\n\r\n1. **`config.py`:** Defines core, global settings:\r\n * `OUTPUT_BASE_DIR`: Default root directory for processed assets.\r\n * `DEFAULT_ASSET_CATEGORY`: Fallback category (\"Texture\", \"Asset\", \"Decal\").\r\n * `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., \"4K\") to pixel dimensions.\r\n * `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `[\"COL\"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.\r\n * `TARGET_FILENAME_PATTERN`: Format string for output filenames.\r\n * `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).\r\n * `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).\r\n * `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.\r\n * `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `[\"NRM\", \"DISP\"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.\r\n * ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).\r\n\r\n2. **`presets/*.json`:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:\r\n * `supplier_name`: Name of the asset source.\r\n * `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `\"target_type\"` (e.g., `\"COL\"`, `\"NRM\"`) and `\"keywords\"` (a list of source filename patterns like `[\"_col*\", \"_color\"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `\"keywords\"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.\r\n * `bit_depth_variants`: Dictionary mapping standard map types (e.g., `\"NRM\"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `\"*_NRM16*.tif\"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.\r\n * `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).\r\n * `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).\r\n * `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.\r\n * `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.\r\n * `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., \"Decal\").\r\n\r\n Use `presets/_template.json` as a starting point for creating new presets.\r\n\r\n## Usage\r\n\r\n### 1. Graphical User Interface (GUI)\r\n\r\n* **Run:**\r\n ```bash\r\n python gui/main_window.py\r\n ```\r\n* **Interface:**\r\n * **Preset Editor Panel (Left):** Allows creating, deleting, loading, editing, and saving presets directly within the main window. Select a preset from the list to load it into the editor tabs.\r\n * **Preset Selector (Top Right):** Select the preset to use for *processing* the current queue.\r\n * **Drag and Drop:** Drag asset ZIP files or folders onto the designated area in the right panel.\r\n * **Preview:** The table shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.\r\n * `Mapped`: Files recognised as standard texture maps (e.g., Color, Normal).\r\n * `Model`: Files recognised as 3D models (e.g., FBX, OBJ).\r\n * `Extra`: Files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation).\r\n * `Unrecognised`: Files that did not match any map, model, or explicit extra pattern.\r\n * `Ignored`: Files intentionally ignored (e.g., superseded by a 16-bit variant).\r\n * `Error`: Indicates an issue during prediction for this file or asset.\r\n * **Options:**\r\n * `Overwrite Existing`: Check to force reprocessing if output already exists.\r\n * `Workers`: Set the number of assets to process concurrently.\r\n * **Controls:**\r\n * `Clear Queue`: Removes all assets from the current processing queue and clears the preview.\r\n * `Start Processing`: Begins the processing pipeline for all added assets.\r\n * `Cancel`: Attempts to stop ongoing processing.\r\n * **Progress Bar:** Shows the overall processing progress.\r\n * **Status Bar:** Displays messages about the current state, errors, or completion.\r\n\r\n### 2. Command-Line Interface (CLI)\r\n\r\n* **Run:**\r\n ```bash\r\n python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]\r\n ```\r\n* **Arguments:**\r\n * `INPUT_PATH`: One or more paths to input ZIP files or folders.\r\n * `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).\r\n * `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.\r\n * `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).\r\n * `--overwrite`: Force reprocessing and overwrite existing output.\r\n * `-v`, `--verbose`: Enable detailed DEBUG level logging.\r\n* **Example:**\r\n ```bash\r\n python main.py \"C:/Downloads/WoodFine001.zip\" \"C:/Downloads/MetalScratched02.zip\" -p Poliigon -o \"G:/Assets/Processed\" --workers 4 --overwrite\r\n ```\r\n\r\n### 3. Directory Monitor (Automated Processing)\r\n\r\n* **Run:**\r\n ```bash\r\n python monitor.py\r\n ```\r\n* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name.\r\n* **Configuration (Environment Variables):**\r\n * `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).\r\n * `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).\r\n * `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).\r\n * `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).\r\n * `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).\r\n * `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).\r\n * `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).\r\n * `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).\r\n* **Output:**\r\n * Logs processing activity to the console.\r\n * Processed assets are created in the `OUTPUT_DIR` following the standard structure.\r\n * The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.\r\n \r\n ### 4. Blender Node Group Creation (Manual Script)\r\n \r\n * **Purpose:** After processing assets with this tool, you can use the provided Blender script to automatically create and configure PBR node groups within a Blender file, leveraging the generated metadata.\r\n * **Script:** `blenderscripts/create_nodegroups.py`\r\n * **Prerequisites:**\r\n * A library of assets processed by this tool, located at a known path.\r\n * A Blender file containing two template node groups named exactly `Template_PBRSET` and `Template_PBRTYPE`. These templates should contain the necessary input/output sockets and internal nodes (e.g., Image Texture nodes labeled by resolution like \"1K\", \"2K\", \"4K\"; Value nodes labeled \"AspectRatioCorrection\" and \"HighestResolution\"; CombineXYZ nodes labeled \"Histogram-[MAPTYPE]\").\r\n * **Configuration (Inside the script):**\r\n * `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located.\r\n * Other variables like template names, node labels, color space mappings, etc., can be adjusted if needed.\r\n * **Run:**\r\n 1. Open your target Blender file (containing the templates).\r\n 2. Open the `blenderscripts/create_nodegroups.py` script in Blender's Text Editor.\r\n 3. Ensure `PROCESSED_ASSET_LIBRARY_ROOT` is correctly set in the script's configuration section.\r\n 4. Click \"Run Script\" (or Alt+P). Check the Blender System Console (Window -> Toggle System Console) for progress and potential errors.\r\n * **Functionality:**\r\n * Scans the configured library path for processed assets (identified by `metadata.json`).\r\n * For each asset, it creates/updates a parent node group (e.g., `PBRSET_AssetName`).\r\n * For each map type (including merged maps like NRMRGH), it creates/updates a child node group using a Base64 encoded name (e.g., `PBRTYPE_[EncodedAssetName_MapType]`) and links it to the corresponding placeholder in the parent group.\r\n * Loads the processed texture images (handling mixed extensions via fallback) into the correct Image Texture nodes within the child groups and sets their color spaces.\r\n * Applies pre-calculated metadata to the parent group:\r\n * Sets the `AspectRatioCorrection` node value based on image dimensions and the `aspect_ratio_change_string`.\r\n * Sets `Histogram-[MAPTYPE]` node values based on `image_stats_1k` from metadata.\r\n * Sets the `HighestResolution` node value (1K=1.0, 2K=2.0, 4K=3.0, 8K=4.0).\r\n * Adds `supplier_name` and `archetype` as asset tags.\r\n * Sets the custom asset preview using the lowest resolution color map.\r\n * Uses an optional manifest file (`[ActiveBlendFileName]_manifest.json`) stored alongside the `.blend` file to track progress and skip already processed maps/resolutions on subsequent runs (requires the `.blend` file to be saved).\r\n \r\n ## Processing Pipeline (Simplified)\r\n \r\n1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.\r\n2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.\r\n3. **Metadata Determination:** Asset name, category, and archetype are determined.\r\n4. **Skip Check:** If output exists and overwrite is off, processing stops here.\r\n5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.\r\n6. **Merging:** Channels are merged according to preset rules and saved.\r\n7. **Metadata Generation:** `metadata.json` is created with all collected information.\r\n8. **Output Organization:** Processed files are moved to the final structured output directory.\r\n9. **Cleanup:** The temporary workspace is removed.\r\n\r\n## Output Structure\r\n\r\nProcessed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`\r\n\r\nEach asset directory typically contains:\r\n* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).\r\n* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).\r\n* Model files (if present in source).\r\n* `metadata.json`: Detailed information about the asset and processing.\r\n* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.\r\n\r\n## Docker\r\n\r\nA `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands."
}
]
}
