Token-based output support - needs testing

Documentation/01_User_Guide/04_Configuration_and_Presets.md

@@ -44,4 +44,14 @@ Preset files define supplier-specific rules for interpreting asset source files.
 
 When processing assets, you must specify which preset to use. The tool then loads the core settings from `config/app_settings.json` and merges them with the rules from the selected preset to determine how to process the input.
 
-A template preset file (`presets/_template.json`) is provided as a base for creating new presets.
+A template preset file (`presets/_template.json`) is provided as a base for creating new presets.
+## Global Output Path Configuration
+
+The structure and naming of the output files generated by the tool are now controlled by two global settings defined exclusively in `config/app_settings.json`:
+
+*   `OUTPUT_DIRECTORY_PATTERN`: Defines the directory structure where processed assets will be saved.
+*   `OUTPUT_FILENAME_PATTERN`: Defines the naming convention for the individual output files within the generated directory.
+
+**Important:** These settings are global and apply to all processing tasks, regardless of the selected preset. They are **not** part of individual preset files and cannot be modified using the Preset Editor. You can view and edit these patterns in the main application preferences (**Edit** -> **Preferences...**).
+
+These patterns use special tokens (e.g., `[assetname]`, `[maptype]`) that are replaced with actual values during processing. For a detailed explanation of how these patterns work together, the available tokens, and examples, please refer to the [Output Structure](./09_Output_Structure.md) section of the User Guide.

Documentation/01_User_Guide/09_Output_Structure.md

@@ -2,20 +2,65 @@
 
 This document describes the directory structure and contents of the processed assets generated by the Asset Processor Tool.
 
-Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`
+Processed assets are saved to a location determined by two global settings defined in `config/app_settings.json`:
 
-*   `<output_base_directory>`: The base output directory configured in `config.py` or specified via CLI/GUI.
-*   `<supplier_name>`: The name of the asset supplier, determined from the preset used.
-*   `<asset_name>`: The name of the processed asset, determined from the source filename based on preset rules.
+*   `OUTPUT_DIRECTORY_PATTERN`: Defines the directory structure *within* the Base Output Directory.
+*   `OUTPUT_FILENAME_PATTERN`: Defines the naming convention for individual files *within* the directory created by `OUTPUT_DIRECTORY_PATTERN`.
+
+These patterns use special tokens (explained below) that are replaced with actual values during processing. You can configure these patterns via the main application preferences (**Edit** -> **Preferences...** -> **Output & Naming** tab). They are global settings and are not part of individual presets.
+
+### Available Tokens
+
+The following tokens can be used in both `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`. Note that some tokens make more sense in one pattern than the other (e.g., `[maptype]` and `[ext]` are typically used in the filename pattern).
+
+*   `[Assettype]`: The type of asset (e.g., `Texture`, `Model`, `Surface`).
+*   `[supplier]`: The supplier name (from the preset, e.g., `Poliigon`).
+*   `[assetname]`: The main asset name (e.g., `RustyMetalPanel`).
+*   `[resolution]`: Texture resolution (e.g., `1k`, `2k`, `4k`).
+*   `[ext]`: The output file extension (e.g., `png`, `jpg`, `exr`). (Primarily for filename pattern)
+*   `[IncrementingValue]` or `[####]`: A numerical value that increments based on existing directories matching the `OUTPUT_DIRECTORY_PATTERN` in the output base path. The number of `#` characters determines the zero-padding (e.g., `[###]` -> `001`, `002`). If `[IncrementingValue]` is used, it defaults to 4 digits of padding (`0001`, `0002`).
+*   `[Date]`: Current date (`YYYYMMDD`).
+*   `[Time]`: Current time (`HHMMSS`).
+*   `[Sha5]`: The first 5 characters of the SHA-256 hash of the original input source file (e.g., the source zip archive).
+*   `[ApplicationPath]`: Absolute path to the application directory.
+*   `[maptype]`: Specific map type (e.g., `Albedo`, `Normal`). (Primarily for filename pattern)
+*   `[dimensions]`: Pixel dimensions (e.g., `2048x2048`).
+*   `[bitdepth]`: Output bit depth (e.g., `8bit`, `16bit`).
+*   `[category]`: Asset category determined by preset rules.
+*   `[archetype]`: Asset archetype determined by preset rules.
+*   `[variant]`: Asset variant identifier determined by preset rules.
+*   `[source_filename]`: The original filename of the source file being processed.
+*   `[source_basename]`: The original filename without the extension.
+*   `[source_dirname]`: The directory containing the original source file.
+
+### Example Output Paths
+
+The final output path is constructed by combining the Base Output Directory (set in Preferences or via CLI) with the results of the two patterns.
+
+**Example 1:**
+
+*   Base Output Directory: `/home/user/ProcessedAssets`
+*   `OUTPUT_DIRECTORY_PATTERN`: `[supplier]/[assetname]/[resolution]`
+*   `OUTPUT_FILENAME_PATTERN`: `[assetname]_[maptype]_[resolution].[ext]`
+*   Resulting Path for an Albedo map: `/home/user/ProcessedAssets/Poliigon/WoodFloor001/4k/WoodFloor001_Albedo_4k.png`
+
+**Example 2:**
+
+*   Base Output Directory: `Output` (relative path)
+*   `OUTPUT_DIRECTORY_PATTERN`: `[Assettype]/[category]/[assetname]`
+*   `OUTPUT_FILENAME_PATTERN`: `[maptype].[ext]`
+*   Resulting Path for a Normal map: `Output/Texture/Wood/WoodFloor001/Normal.exr`
+
+The `<output_base_directory>` (the root folder where processing output starts) is configured separately via the GUI (**Edit** -> **Preferences...** -> **Output & Naming** tab -> **Base Output Directory**) or the `--output` CLI argument. The `OUTPUT_DIRECTORY_PATTERN` defines the structure *within* this base directory, and `OUTPUT_FILENAME_PATTERN` defines the filenames within that structure.
 
 ## Contents of Each Asset Directory
 
 Each asset directory contains the following:
 
-*   Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`). These are the resized, format-converted, and bit-depth adjusted texture files.
-*   Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`). These are maps created by combining channels from different source maps based on the configured merge rules.
+*   Processed texture maps (e.g., `WoodFloor_Albedo_4k.png`, `MetalPanel_Normal_2k.exr`). The exact filenames depend on the `OUTPUT_FILENAME_PATTERN`. These are the resized, format-converted, and bit-depth adjusted texture files.
+*   Merged texture maps (e.g., `WoodFloor_Combined_4k.png`). The exact filenames depend on the `OUTPUT_FILENAME_PATTERN`. These are maps created by combining channels from different source maps based on the configured merge rules.
 *   Model files (if present in the source asset).
 *   `metadata.json`: A JSON file containing detailed information about the asset and the processing that was performed. This includes details about the maps, resolutions, formats, bit depths, merged map details, calculated image statistics, aspect ratio change information, asset category and archetype, the source preset used, and a list of ignored source files. This file is intended for use by downstream tools or scripts (like the Blender integration scripts).
-*   `Extra/` (subdirectory): Contains source files that were not classified as maps or models but were explicitly marked to be moved to the extra directory based on preset rules (e.g., previews, documentation files).
+*   `EXTRA/` (subdirectory): Contains source files not classified as maps or models but marked as "EXTRA" by preset rules (e.g., previews, documentation). These files are placed in an `EXTRA` folder *within* the directory generated by `OUTPUT_DIRECTORY_PATTERN`.
 *   `Unrecognised/` (subdirectory): Contains source files that were not classified as maps, models, or explicitly marked as extra, and were not ignored.
 *   `Ignored/` (subdirectory): Contains source files that were explicitly ignored during processing (e.g., an 8-bit Normal map when a 16-bit variant exists and is prioritized).

Documentation/02_Developer_Guide/04_Configuration_System_and_Presets.md

@@ -6,7 +6,7 @@ This document provides technical details about the configuration system and the
 
 The tool utilizes a two-tiered configuration system managed by the `configuration.py` module:
 
-1.  **Application Settings (`config/app_settings.json`):** This JSON file defines the core global default settings, constants, and rules that apply generally across different asset sources (e.g., default output paths, standard image resolutions, map merge rules, output format rules, Blender paths, `FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`).
+1.  **Application Settings (`config/app_settings.json`):** This JSON file defines the core global default settings, constants, and rules that apply generally across different asset sources (e.g., the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`, standard image resolutions, map merge rules, output format rules, Blender paths, `FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`). See the [User Guide: Output Structure](../01_User_Guide/09_Output_Structure.md#available-tokens) for a list of available tokens for these patterns.
 2.  **LLM Settings (`config/llm_settings.json`):** This JSON file contains settings specifically related to the LLM predictor, such as the API endpoint, model name, prompt template, and examples. These settings can be edited through the GUI using the `LLMEditorWidget`.
 3.  **Preset Files (`Presets/*.json`):** These JSON files define supplier-specific rules and overrides. They contain patterns to interpret filenames, classify map types, handle variants, define naming conventions, and specify other source-specific behaviors.
 
@@ -28,7 +28,7 @@ The `Configuration` class is central to the new configuration system. It is resp
     *   It first loads the base application settings from `config/app_settings.json`.
     *   It then loads the LLM-specific settings from `config/llm_settings.json`.
     *   Finally, it loads the specified preset JSON file from the `Presets/` directory.
-*   **Merging & Access:** The base settings from `app_settings.json` are merged with the preset rules. LLM settings are stored separately. All settings are accessible via instance properties (e.g., `config.target_filename_pattern`, `config.llm_endpoint_url`). Preset values generally override the base settings where applicable.
+*   **Merging & Access:** The base settings from `app_settings.json` are merged with the preset rules. LLM settings are stored separately. Most settings are accessible via instance properties (e.g., `config.llm_endpoint_url`). Preset values generally override the base settings where applicable. **Exception:** The `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN` are loaded *only* from `app_settings.json` and are accessed via `config.output_directory_pattern` and `config.output_filename_pattern` respectively; they are not defined in or overridden by presets.
 *   **Validation (`_validate_configs`):** Performs basic structural validation on the loaded settings (base, LLM, and preset), checking for the presence of required keys and basic data types. Logs warnings for missing optional LLM keys.
 *   **Regex Compilation (`_compile_regex_patterns`):** Compiles regex patterns defined in the merged configuration (from base settings and the preset) for performance. Compiled regex objects are stored as instance attributes (e.g., `self.compiled_map_keyword_regex`).
 *   **LLM Settings Access:** The `Configuration` class provides direct property access (e.g., `config.llm_endpoint_url`, `config.llm_api_key`, `config.llm_model_name`, `config.llm_temperature`, `config.llm_request_timeout`, `config.llm_predictor_prompt`, `config.get_llm_examples()`) to allow components like the `LLMPredictionHandler` to easily access the necessary LLM configuration values loaded from `config/llm_settings.json`.
@@ -47,7 +47,7 @@ The GUI provides dedicated editors for modifying configuration files:
 The GUI includes a dedicated editor for modifying the `config/app_settings.json` file. This is implemented in `gui/config_editor_dialog.py`.
 
 *   **Purpose:** Provides a user-friendly interface for viewing and editing the core application settings defined in `app_settings.json`.
-*   **Implementation:** The dialog loads the JSON content of `app_settings.json`, presents it in a tabbed layout ("General", "Output & Naming", etc.) using standard GUI widgets mapped to the JSON structure, and saves the changes back to the file. It supports editing basic fields, tables for definitions (`FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`), and a list/detail view for merge rules (`MAP_MERGE_RULES`). The definitions tables include dynamic color editing features.
+*   **Implementation:** The dialog loads the JSON content of `app_settings.json`, presents it in a tabbed layout ("General", "Output & Naming", etc.) using standard GUI widgets mapped to the JSON structure, and saves the changes back to the file. The "Output & Naming" tab specifically handles the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`. It supports editing basic fields, tables for definitions (`FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`), and a list/detail view for merge rules (`MAP_MERGE_RULES`). The definitions tables include dynamic color editing features.
 *   **Limitations:** Currently, editing complex fields like `IMAGE_RESOLUTIONS` or the full details of `MAP_MERGE_RULES` via the UI is not fully supported.
 *   **Note:** Changes made through the `ConfigEditorDialog` are written directly to `config/app_settings.json` (using `save_base_config`) but require an application restart to be loaded and applied by the `Configuration` class during processing.
 

Documentation/02_Developer_Guide/05_Processing_Pipeline.md

@@ -54,6 +54,7 @@ The pipeline steps are:
     *   Writes data to `metadata.json` in the temporary workspace.
 
 9.  **Output Organization (`_organize_output_files`)**:
+*   Determines the final output directory using the global `OUTPUT_DIRECTORY_PATTERN` and the final filename using the global `OUTPUT_FILENAME_PATTERN` (both from the `Configuration` object). The `utils.path_utils` module combines these with the base output directory and asset-specific data (like asset name, map type, resolution, etc.) to construct the full path for each file.
     *   Creates the final structured output directory (`<output_base_dir>/<supplier_name>/<asset_name>/`), using the supplier name from the `SourceRule`.
     *   Moves processed maps, merged maps, models, metadata, and other classified files from the temporary workspace to the final output directory.