Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Config Updates - User settings - Saving Methods

Browse Source
This commit is contained in:
Rusfort 2025-05-13 10:32:19 +02:00
parent 383e904e1a
commit dec5d7d27f
14 changed files with 1279 additions and 974 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

100
Documentation/02_Developer_Guide/04_Configuration_System_and_Presets.md
View File

@ -2,13 +2,21 @@
This document provides technical details about the configuration system and the structure of preset files for developers working on the Asset Processor Tool.
## Configuration Flow
## Configuration System Overview
The tool utilizes a two-tiered configuration system managed by the `configuration.py` module:
The tool's configuration is managed by the `configuration.py` module and loaded from several JSON files, providing a layered approach for defaults, user overrides, definitions, and source-specific presets.
1. **Application Settings (`config/app_settings.json`):** This JSON file defines the core global default settings, constants, and rules that apply generally across different asset sources (e.g., the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`, standard image resolutions, map merge rules, output format rules, Blender paths, `FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`). See the [User Guide: Output Structure](../01_User_Guide/09_Output_Structure.md#available-tokens) for a list of available tokens for these patterns.
* **`FILE_TYPE_DEFINITIONS` Enhancements:**
* **`keybind` Property:** Each file type object within `FILE_TYPE_DEFINITIONS` can now optionally include a `keybind` property. This property accepts a single character string (e.g., `"C"`, `"R"`) representing the keyboard key. In the GUI, this key (typically combined with `Ctrl`, or standalone like `F2` for asset naming) is used as a shortcut to set or toggle the corresponding file type for selected items in the Preview Table.
### Configuration Files
1. **Application Settings (`config/app_settings.json`):** This JSON file defines the core global default settings, constants, and rules that apply generally across different asset sources (e.g., the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`, standard image resolutions, map merge rules, output format rules, Blender paths). See the [User Guide: Output Structure](../01_User_Guide/09_Output_Structure.md#available-tokens) for a list of available tokens for these patterns.
* *Note:* `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` are no longer stored here; they have been moved to dedicated files.
2. **User Settings (`config/user_settings.json`):** This optional JSON file allows users to override specific settings defined in `config/app_settings.json`. If this file exists, its values for corresponding keys will take precedence over the base application settings. This file is primarily managed through the GUI's Configuration Editor.
3. **Asset Type Definitions (`config/asset_type_definitions.json`):** This dedicated JSON file contains the definitions for different asset types (e.g., Surface, Model, Decal), including their descriptions, colors, and examples.
4. **File Type Definitions (`config/file_type_definitions.json`):** This dedicated JSON file contains the definitions for different file types (specifically texture maps and models), including descriptions, colors, examples, standard aliases, bit depth rules, grayscale flags, and GUI keybinds.
* **`keybind` Property:** Each file type object within `FILE_TYPE_DEFINITIONS` can optionally include a `keybind` property. This property accepts a single character string (e.g., `"C"`, `"R"`) representing the keyboard key. In the GUI, this key (typically combined with `Ctrl`, or standalone like `F2` for asset naming) is used as a shortcut to set or toggle the corresponding file type for selected items in the Preview Table.
*Example:*
```json
"MAP_COL": {
@ -21,7 +29,7 @@ The tool utilizes a two-tiered configuration system managed by the `configuratio
"keybind": "C"
},
```
* **New File Type `MAP_GLOSS`:** A new standard file type, `MAP_GLOSS`, has been added. It is typically configured as follows:
* **New File Type `MAP_GLOSS`:** A standard file type, `MAP_GLOSS`, is defined here.
*Example:*
```json
"MAP_GLOSS": {
@ -35,10 +43,84 @@ The tool utilizes a two-tiered configuration system managed by the `configuratio
}
```
Note: The `keybind` "R" for `MAP_GLOSS` is often shared with `MAP_ROUGH` to allow toggling between them.
2. **LLM Settings (`config/llm_settings.json`):** This JSON file contains settings specifically related to the LLM predictor, such as the API endpoint, model name, prompt template, and examples. These settings can be edited through the GUI using the `LLMEditorWidget`.
3. **Preset Files (`Presets/*.json`):** These JSON files define supplier-specific rules and overrides. They contain patterns to interpret filenames, classify map types, handle variants, define naming conventions, and specify other source-specific behaviors.
The `configuration.py` module contains the `Configuration` class (for loading/merging settings for processing) and standalone functions like `load_base_config()` (for accessing `app_settings.json` directly) and `save_llm_config()` / `save_base_config()` (for writing settings back to files). Note that the old `config.py` file has been deleted.
5. **LLM Settings (`config/llm_settings.json`):** This JSON file contains settings specifically related to the LLM predictor, such as the API endpoint, model name, prompt template, and examples. These settings can be edited through the GUI using the `LLMEditorWidget`.
6. **Preset Files (`Presets/*.json`):** These JSON files define supplier-specific rules and overrides. They contain patterns to interpret filenames, classify map types, handle variants, define naming conventions, and specify other source-specific behaviors. Preset settings override values from `app_settings.json` and `user_settings.json` where applicable.
### Configuration Loading and Access
The `configuration.py` module contains the `Configuration` class and standalone functions for loading and saving settings.
* **`Configuration` Class:** This is the primary class used by the processing engine and other core components. When initialized with a `preset_name`, it loads settings in the following order, with later files overriding earlier ones for shared keys:
1. `config/app_settings.json` (Base Defaults)
2. `config/user_settings.json` (User Overrides - if exists)
3. `config/asset_type_definitions.json` (Asset Type Definitions)
4. `config/file_type_definitions.json` (File Type Definitions)
5. `config/llm_settings.json` (LLM Settings)
6. `Presets/{preset_name}.json` (Preset Overrides)
The loaded settings are merged into internal dictionaries, and most are accessible via instance properties (e.g., `config.output_base_dir`, `config.llm_endpoint_url`, `config.get_asset_type_definitions()`). Regex patterns defined in the merged configuration are pre-compiled for performance.
* **`load_base_config()` function:** This standalone function is primarily used by the GUI for initial setup and displaying default/user-overridden settings before a specific preset is selected. It loads and merges the following files:
1. `config/app_settings.json`
2. `config/user_settings.json` (if exists)
3. `config/asset_type_definitions.json`
4. `config/file_type_definitions.json`
It returns a single dictionary containing the combined settings and definitions.
* **Saving Functions:**
* `save_base_config(settings_dict)`: Saves the provided dictionary to `config/app_settings.json`. (Used less frequently now for user-driven saves).
* `save_user_config(settings_dict)`: Saves the provided dictionary to `config/user_settings.json`. Used by `ConfigEditorDialog`.
* `save_llm_config(settings_dict)`: Saves the provided dictionary to `config/llm_settings.json`. Used by `LLMEditorWidget`.
## Supplier Management (`config/suppliers.json`)
A file, `config/suppliers.json`, is used to store a persistent list of known supplier names. This file is a simple JSON array of strings.
* **Purpose:** Provides a list of suggestions for the "Supplier" field in the GUI's Unified View, enabling auto-completion.
* **Management:** The GUI's `SupplierSearchDelegate` is responsible for loading this list on startup, adding new, unique supplier names entered by the user, and saving the updated list back to the file.
## GUI Configuration Editors
The GUI provides dedicated editors for modifying configuration files:
* **`ConfigEditorDialog` (`gui/config_editor_dialog.py`):** Edits user-configurable application settings.
* **`LLMEditorWidget` (`gui/llm_editor_widget.py`):** Edits the LLM-specific settings.
### `ConfigEditorDialog` (`gui/config_editor_dialog.py`)
The GUI includes a dedicated editor for modifying user-configurable settings. This is implemented in `gui/config_editor_dialog.py`.
* **Purpose:** Provides a user-friendly interface for viewing the effective application settings (defaults + user overrides + definitions) and editing the user-specific overrides.
* **Implementation:** The dialog loads the effective settings using `load_base_config()`. It presents relevant settings in a tabbed layout ("General", "Output & Naming", etc.). When saving, it now performs a **granular save**: it loads the current content of `config/user_settings.json`, identifies only the settings that were changed by the user during the current dialog session (by comparing against the initial state), updates only those specific values in the loaded `user_settings.json` content, and saves the modified content back to `config/user_settings.json` using `save_user_config()`. This preserves any other settings in `user_settings.json` that were not touched. The dialog displays definitions from `asset_type_definitions.json` and `file_type_definitions.json` but does not save changes to these files.
* **Limitations:** Currently, editing complex fields like `IMAGE_RESOLUTIONS` or the full details of `MAP_MERGE_RULES` via the UI is not fully supported for saving to `user_settings.json`.
### `LLMEditorWidget` (`gui/llm_editor_widget.py`)
* **Purpose:** Provides a user-friendly interface for viewing and editing the LLM settings defined in `config/llm_settings.json`.
* **Implementation:** Uses tabs for "Prompt Settings" and "API Settings". Allows editing the prompt, managing examples, and configuring API details. When saving, it also performs a **granular save**: it loads the current content of `config/llm_settings.json`, identifies only the settings changed by the user in the current session, updates only those values, and saves the modified content back to `config/llm_settings.json` using `configuration.save_llm_config()`.
## Preset File Structure (`Presets/*.json`)
Preset files are the primary way to adapt the tool to new asset sources. Developers should use `Presets/_template.json` as a starting point. Key fields include:
* `supplier_name`: The name of the asset source (e.g., `"Poliigon"`). Used for output directory naming.
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a specific file type. The `target_type` for this mapping **must** be a key from the `FILE_TYPE_DEFINITIONS` now located in `config/file_type_definitions.json`.
* `target_type`: The specific file type key from `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_NORM_GL"`, `"MAP_RGH"`). This replaces previous alias-based systems. The common aliases like "COL" or "NRM" are now derived from the `standard_type` property within `FILE_TYPE_DEFINITIONS` but are not used directly for `target_type`.
* `keywords`: A list of filename patterns (regex or fnmatch-style wildcards) used to identify this map type. The order of keywords within this list, and the order of dictionaries in the `map_type_mapping` list, determines the priority for assigning variant suffixes (`-1`, `-2`, etc.) when multiple files match the same `target_type`.
* `bit_depth_variants`: A dictionary mapping standard map types (e.g., `"NRM"`) to a pattern identifying its high bit-depth variant (e.g., `"*_NRM16*.tif"`). Files matching these patterns are prioritized over their standard counterparts.
* `map_bit_depth_rules`: Defines how to handle the bit depth of source maps. Can specify a default behavior (`"respect"` or `"force_8bit"`) and overrides for specific map types.
* `model_patterns`: A list of regex patterns to identify model files (e.g., `".*\\.fbx"`, `".*\\.obj"`).
* `move_to_extra_patterns`: A list of regex patterns for files that should be moved directly to the `Extra/` output subdirectory without further processing.
* `source_naming_convention`: Rules for extracting the base asset name and potentially the archetype from source filenames or directory structures (e.g., using separators and indices).
* `asset_category_rules`: Keywords or patterns used to determine the asset category (e.g., identifying `"Decal"` based on keywords).
* `archetype_rules`: Keywords or patterns used to determine the asset archetype (e.g., identifying `"Wood"` or `"Metal"`).
Careful definition of these patterns and rules, especially the regex in `map_type_mapping`, `bit_depth_variants`, `model_patterns`, and `move_to_extra_patterns`, is essential for correct asset processing.
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how configuration data is handled and passed within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
## Supplier Management (`config/suppliers.json`)

98
Presets/Poliigon.json
View File

@ -25,102 +25,10 @@
"*.pdf",
"*.url",
"*.htm*",
"*_Fabric.*"
],
"map_type_mapping": [
{
"target_type": "MAP_COL",
"keywords": [
"COLOR*",
"COL",
"DIFFUSE",
"DIF",
"ALBEDO"
]
},
{
"target_type": "MAP_NRM",
"keywords": [
"NORMAL*",
"NORM*",
"NRM*"
]
},
{
"target_type": "MAP_ROUGH",
"keywords": [
"ROUGHNESS",
"ROUGH"
]
},
{
"target_type": "MAP_GLOSS",
"keywords": [
"GLOSS"
],
"is_gloss_source": true
},
{
"target_type": "MAP_AO",
"keywords": [
"AMBIENTOCCLUSION",
"AO"
]
},
{
"target_type": "MAP_DISP",
"keywords": [
"DISPLACEMENT",
"DISP",
"HEIGHT",
"BUMP"
]
},
{
"target_type": "MAP_REFL",
"keywords": [
"REFLECTION",
"REFL",
"SPECULAR",
"SPEC"
]
},
{
"target_type": "MAP_SSS",
"keywords": [
"SSS",
"SUBSURFACE*"
]
},
{
"target_type": "MAP_FUZZ",
"keywords": [
"FUZZ"
]
},
{
"target_type": "MAP_IDMAP",
"keywords": [
"IDMAP"
]
},
{
"target_type": "MAP_MASK",
"keywords": [
"OPAC*",
"TRANSP*",
"MASK*",
"ALPHA*"
]
},
{
"target_type": "MAP_METAL",
"keywords": [
"METAL*",
"METALLIC"
]
}
"*_Fabric.*",
"*_Albedo*"
],
"map_type_mapping": [],
"asset_category_rules": {
"model_patterns": [
"*.fbx",

107
ProjectNotes/ConfigurationRefactoringPlan.md Normal file
View File

@ -0,0 +1,107 @@
# Configuration System Refactoring Plan
This document outlines the plan for refactoring the configuration system of the Asset Processor Tool.
## Overall Goals
1. **Decouple Definitions:** Separate `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` from the main `config/app_settings.json` into dedicated files.
2. **Introduce User Overrides:** Allow users to override base settings via a new `config/user_settings.json` file.
3. **Improve GUI Saving:** (Lower Priority) Make GUI configuration saving more targeted to avoid overwriting unrelated settings when saving changes from `ConfigEditorDialog` or `LLMEditorWidget`.
## Proposed Plan Phases
**Phase 1: Decouple Definitions**
1. **Create New Definition Files:**
* Create `config/asset_type_definitions.json`.
* Create `config/file_type_definitions.json`.
2. **Migrate Content:**
* Move `ASSET_TYPE_DEFINITIONS` object from `config/app_settings.json` to `config/asset_type_definitions.json`.
* Move `FILE_TYPE_DEFINITIONS` object from `config/app_settings.json` to `config/file_type_definitions.json`.
3. **Update `configuration.py`:**
* Add constants for new definition file paths.
* Modify `Configuration` class to load these new files.
* Update property methods (e.g., `get_asset_type_definitions`, `get_file_type_definitions_with_examples`) to use data from the new definition dictionaries.
* Adjust validation (`_validate_configs`) as needed.
4. **Update GUI & `load_base_config()`:**
* Modify `load_base_config()` to load and return a combined dictionary including `app_settings.json` and the two new definition files.
* Update GUI components relying on `load_base_config()` to ensure they receive the necessary definition data.
**Phase 2: Implement User Overrides**
1. **Define `user_settings.json`:**
* Establish `config/user_settings.json` for user-specific overrides, mirroring parts of `app_settings.json`.
2. **Update `configuration.py` Loading:**
* In `Configuration.__init__`, load `app_settings.json`, then definition files, then attempt to load and deep merge `user_settings.json` (user settings override base).
* Load presets *after* the base+user merge (presets override combined base+user).
* Modify `load_base_config()` to also load and merge `user_settings.json` after `app_settings.json`.
3. **Update GUI Editors:**
* Modify `ConfigEditorDialog` to load the effective settings (base+user) but save changes *only* to `config/user_settings.json`.
* `LLMEditorWidget` continues targeting `llm_settings.json`.
**Phase 3: Granular GUI Saving (Lower Priority)**
1. **Refactor Saving Logic:**
* In `ConfigEditorDialog` and `LLMEditorWidget`:
* Load the current target file (`user_settings.json` or `llm_settings.json`).
* Identify specific setting(s) changed by the user in the GUI session.
* Update only those specific key(s) in the loaded dictionary.
* Write the entire modified dictionary back to the target file, preserving untouched settings.
## Proposed File Structure & Loading Flow
```mermaid
graph LR
subgraph Config Files
A[config/asset_type_definitions.json]
B[config/file_type_definitions.json]
C[config/app_settings.json (Base Defaults)]
D[config/user_settings.json (User Overrides)]
E[config/llm_settings.json]
F[config/suppliers.json]
G[Presets/*.json]
end
subgraph Code
H[configuration.py]
I[GUI]
J[Processing Engine / Pipeline]
K[LLM Handlers]
end
subgraph Loading Flow (Configuration Class)
L(Load Asset Types) --> H
M(Load File Types) --> H
N(Load Base Settings) --> P(Merge Base + User)
O(Load User Settings) --> P
P --> R(Merge Preset Overrides)
Q(Load LLM Settings) --> H
R --> T(Final Config Object)
G -- Load Preset --> R
H -- Contains --> T
end
subgraph Loading Flow (GUI - load_base_config)
L2(Load Asset Types) --> U(Return Merged Defaults + Defs)
M2(Load File Types) --> U
N2(Load Base Settings) --> V(Merge Base + User)
O2(Load User Settings) --> V
V --> U
I -- Calls --> U
end
T -- Used by --> J
T -- Used by --> K
I -- Edits --> D
I -- Edits --> E
I -- Manages --> F
style A fill:#f9f,stroke:#333,stroke-width:2px
style B fill:#f9f,stroke:#333,stroke-width:2px
style C fill:#ccf,stroke:#333,stroke-width:2px
style D fill:#9cf,stroke:#333,stroke-width:2px
style E fill:#ccf,stroke:#333,stroke-width:2px
style F fill:#9cf,stroke:#333,stroke-width:2px
style G fill:#ffc,stroke:#333,stroke-width:2px

96
ProjectNotes/MAP_Prefix_Enforcement_Plan.md
View File

@ -1,96 +0,0 @@
# Plan: Enforcing "MAP_" Prefix for Internal Processing and Standard Type for Output Naming
**Date:** 2025-05-13
**I. Goal:**
The primary goal is to ensure that for all internal processing, the system *exclusively* uses `FileRule.item_type` values that start with the "MAP_" prefix (e.g., "MAP_COL", "MAP_NRM"). The "standard type" (e.g., "COL", "NRM") associated with these "MAP_" types (as defined in `config/app_settings.json`) should *only* be used during the file saving stages for output naming. Any `FileRule` whose `item_type` does not start with "MAP_" (and isn't a special type like "EXTRA" or "MODEL") should be skipped by the relevant map processing stages.
**II. Current State Analysis Summary:**
* **Output Naming:** The use of "standard type" for output filenames via the `get_filename_friendly_map_type` utility in `SaveVariantsStage` and `OutputOrganizationStage` is **correct** and already meets the requirement.
* **Internal "MAP_" Prefix Usage:**
* Some stages like `GlossToRoughConversionStage` correctly check for "MAP_" prefixes (e.g., `processing_map_type.startswith("MAP_GLOSS")`).
* Other stages like `RegularMapProcessorStage` and `MergedTaskProcessorStage` (and its helpers) implicitly expect "MAP_" prefixed types for their internal regex-based logic but lack explicit checks to skip items if the prefix is missing.
* Stages like `AlphaExtractionToMaskStage` and `NormalMapGreenChannelStage` currently use non-"MAP_" prefixed "standard types" (e.g., "NORMAL", "ALBEDO") when reading from `context.processed_maps_details` for their decision-making logic.
* The `PrepareProcessingItemsStage` adds `FileRule`s to the processing queue without filtering based on the "MAP_" prefix in `item_type`.
* **Data Consistency in `AssetProcessingContext`:**
* `FileRule.item_type` is the field that should hold the "MAP_" prefixed type from the initial rule generation.
* `context.processed_maps_details` entries can contain various map type representations:
* `map_type`: Often stores the "standard type" (e.g., "Roughness", "MASK", "NORMAL").
* `processing_map_type` / `internal_map_type`: Generally seem to store the "MAP_" prefixed type. This needs to be consistent.
* **Configuration (`config/app_settings.json`):**
* `FILE_TYPE_DEFINITIONS` correctly use "MAP_" prefixed keys.
* `MAP_MERGE_RULES` need to be reviewed to ensure their `output_map_type` and input map types are "MAP_" prefixed.
**III. Proposed Changes (Code Identification & Recommendations):**
**A. Enforce "MAP_" Prefix for Processing Items (Skipping Logic):**
The core requirement is that processing stages should skip `FileRule` items if their `item_type` doesn't start with "MAP_".
1. **`RegularMapProcessorStage` (`processing/pipeline/stages/regular_map_processor.py`):**
* **Identify:** In the `execute` method, `initial_internal_map_type` is derived from `file_rule.item_type_override` or `file_rule.item_type`.
* **Recommend:** Add an explicit check after determining `initial_internal_map_type`. If `initial_internal_map_type` does not start with `"MAP_"`, the stage should log a warning, set the `result.status` to "Skipped (Invalid Type)" or similar, and return `result` early, effectively skipping processing for this item.
2. **`MergedTaskProcessorStage` (`processing/pipeline/stages/merged_task_processor.py`):**
* **Identify:** This stage processes `MergeTaskDefinition`s. The definitions for these tasks (input types, output type) come from `MAP_MERGE_RULES` in `config/app_settings.json`. The stage uses `required_map_type_from_rule` for its inputs.
* **Recommend:**
* **Configuration First:** Review all entries in `MAP_MERGE_RULES` in `config/app_settings.json`.
* Ensure the `output_map_type` for each rule (e.g., "MAP_NRMRGH") starts with "MAP_".
* Ensure all map type values within the `inputs` dictionary (e.g., `"R": "MAP_NRM"`) start with "MAP_".
* **Stage Logic:** In the `execute` method, when iterating through `merge_inputs_config.items()`, check if `required_map_type_from_rule` starts with `"MAP_"`. If not, log a warning and either:
* Skip loading/processing this specific input channel (potentially using its fallback if the overall merge can still proceed).
* Or, if a non-"MAP_" input is critical, fail the entire merge task for this asset.
* The helper `_apply_in_memory_transformations` already uses regex expecting "MAP_" prefixes; this will naturally fail or misbehave if inputs are not "MAP_" prefixed, reinforcing the need for the check above.
**B. Standardize Map Type Fields and Usage in `context.processed_maps_details`:**
Ensure consistency in how "MAP_" prefixed types are stored and accessed within `context.processed_maps_details` for internal logic (not naming).
1. **Recommendation:** Establish a single, consistent field name within `context.processed_maps_details` to store the definitive "MAP_" prefixed internal map type (e.g., `internal_map_type` or `processing_map_type`). All stages that perform logic based on the specific *kind* of map (e.g., transformations, source selection) should read from this standardized field. The `map_type` field can continue to store the "standard type" (e.g., "Roughness") primarily for informational/metadata purposes if needed, but not for core processing logic.
2. **`AlphaExtractionToMaskStage` (`processing/pipeline/stages/alpha_extraction_to_mask.py`):**
* **Identify:**
* Checks for existing MASK map using `file_rule.map_type == "MASK"`. (Discrepancy: `FileRule` uses `item_type`).
* Searches for suitable source maps using `details.get('map_type') in self.SUITABLE_SOURCE_MAP_TYPES` where `SUITABLE_SOURCE_MAP_TYPES` are standard types like "ALBEDO".
* When adding new details, it sets `map_type: "MASK"` and the new `FileRule` gets `item_type="MAP_MASK"`.
* **Recommend:**
* Change the check for an existing MASK map to `file_rule.item_type == "MAP_MASK"`.
* Modify the source map search to use the standardized "MAP_" prefixed field from `details` (e.g., `details.get('internal_map_type')`) and update `SUITABLE_SOURCE_MAP_TYPES` to be "MAP_" prefixed (e.g., "MAP_COL", "MAP_ALBEDO").
* When adding new details for the created MASK map to `context.processed_maps_details`, ensure the standardized "MAP_" prefixed field is set to "MAP_MASK", and `map_type` (if kept) is "MASK".
3. **`NormalMapGreenChannelStage` (`processing/pipeline/stages/normal_map_green_channel.py`):**
* **Identify:** Checks `map_details.get('map_type') == "NORMAL"`.
* **Recommend:** Change this check to use the standardized "MAP_" prefixed field from `map_details` (e.g., `map_details.get('internal_map_type')`) and verify if it `startswith("MAP_NRM")`.
4. **`GlossToRoughConversionStage` (`processing/pipeline/stages/gloss_to_rough_conversion.py`):**
* **Identify:** This stage already uses `processing_map_type.startswith("MAP_GLOSS")` and updates `processing_map_type` to "MAP_ROUGH" in `map_details`. It also updates the `FileRule.item_type` correctly.
* **Recommend:** This stage is largely consistent. Ensure the field it reads/writes (`processing_map_type`) aligns with the chosen standardized "MAP_" prefixed field for `processed_maps_details`.
**C. Review Orchestration Logic (Conceptual):**
* When the orchestrator populates `context.processed_maps_details` after stages like `SaveVariantsStage`, ensure it stores the "MAP_" prefixed `internal_map_type` (from `SaveVariantsInput`) into the chosen standardized field in `processed_maps_details`.
**IV. Testing Recommendations:**
* Create test cases with `AssetRule`s containing `FileRule`s where `item_type` is intentionally set to a non-"MAP_" prefixed value (e.g., "COLOR_MAP", "TEXTURE_ROUGH"). Verify that `RegularMapProcessorStage` skips these.
* Modify `MAP_MERGE_RULES` in a test configuration:
* Set an `output_map_type` to a non-"MAP_" value.
* Set an input map type (e.g., for channel "R") to a non-"MAP_" value.
* Verify that `MergedTaskProcessorStage` correctly handles these (e.g., fails the task, skips the input, logs warnings).
* Test `AlphaExtractionToMaskStage`:
* With an existing `FileRule` having `item_type="MAP_MASK"` to ensure extraction is skipped.
* With source maps having "MAP_COL" (with alpha) as their `internal_map_type` in `processed_maps_details` to ensure they are correctly identified as sources.
* Test `NormalMapGreenChannelStage` with a normal map having "MAP_NRM" as its `internal_map_type` in `processed_maps_details` to ensure it's processed.
* Verify that output filenames continue to use the "standard type" (e.g., "COL", "ROUGH", "NRM") correctly.
**V. Mermaid Diagram (Illustrative Flow for `FileRule` Processing):**
```mermaid
graph TD
A[AssetRule with FileRules] --> B{FileRuleFilterStage};
B -- files_to_process --> C{PrepareProcessingItemsStage};
C -- processing_items (FileRule) --> D{PipelineOrchestrator};
D -- FileRule --> E(RegularMapProcessorStage);
E --> F{Check FileRule.item_type};
F -- Starts with "MAP_"? --> G[Process Map];
F -- No --> H[Skip Map / Log Warning];
G --> I[...subsequent stages...];
H --> I;

72
ProjectNotes/PipelineRefactoringPlan.md
View File

@ -1,72 +0,0 @@
# Processing Pipeline Refactoring Plan
## 1. Problem Summary
The current processing pipeline, particularly the `IndividualMapProcessingStage`, exhibits maintainability challenges:
* **High Complexity:** The stage handles too many responsibilities (loading, merging, transformations, scaling, saving).
* **Duplicated Logic:** Image transformations (Gloss-to-Rough, Normal Green Invert) are duplicated within the stage instead of relying solely on dedicated stages or being handled consistently.
* **Tight Coupling:** Heavy reliance on the large, mutable `AssetProcessingContext` object creates implicit dependencies and makes isolated testing difficult.
## 2. Refactoring Goals
* Improve code readability and understanding.
* Enhance maintainability by localizing changes and removing duplication.
* Increase testability through smaller, focused components with clear interfaces.
* Clarify data dependencies between pipeline stages.
* Adhere more closely to the Single Responsibility Principle (SRP).
## 3. Proposed New Pipeline Stages
Replace the existing `IndividualMapProcessingStage` with the following sequence of smaller, focused stages, executed by the `PipelineOrchestrator` for each processing item:
1. **`PrepareProcessingItemsStage`:**
* **Responsibility:** Identifies and lists all items (`FileRule`, `MergeTaskDefinition`) to be processed from the main context.
* **Output:** Updates `context.processing_items`.
2. **`RegularMapProcessorStage`:** (Handles `FileRule` items)
* **Responsibility:** Loads source image, determines internal map type (with suffix), applies relevant transformations (Gloss-to-Rough, Normal Green Invert), determines original metadata.
* **Output:** `ProcessedRegularMapData` object containing transformed image data and metadata.
3. **`MergedTaskProcessorStage`:** (Handles `MergeTaskDefinition` items)
* **Responsibility:** Loads input images, applies transformations to inputs, handles fallbacks/resizing, performs merge operation.
* **Output:** `ProcessedMergedMapData` object containing merged image data and metadata.
4. **`InitialScalingStage`:** (Optional)
* **Responsibility:** Applies configured scaling (e.g., POT downscale) to the processed image data received from the previous stage.
* **Output:** Scaled image data.
5. **`SaveVariantsStage`:**
* **Responsibility:** Takes the final processed (and potentially scaled) image data and orchestrates saving variants using the `save_image_variants` utility.
* **Output:** List of saved file details (`saved_files_details`).
## 4. Proposed Data Flow
* **Input/Output Objects:** Key stages (`RegularMapProcessor`, `MergedTaskProcessor`, `InitialScaling`, `SaveVariants`) will use specific Input and Output dataclasses for clearer interfaces.
* **Orchestrator Role:** The `PipelineOrchestrator` manages the overall flow. It calls stages, passes necessary data (extracting image data references and metadata from previous stage outputs to create inputs for the next), receives output objects, and integrates final results (like saved file details) back into the main `AssetProcessingContext`.
* **Image Data Handling:** Large image arrays (`np.ndarray`) are passed primarily via stage return values (Output objects) and used as inputs to subsequent stages, managed by the Orchestrator. They are not stored long-term in the main `AssetProcessingContext`.
* **Main Context:** The `AssetProcessingContext` remains for overall state (rules, paths, configuration access, final status tracking) and potentially for simpler stages with minimal side effects.
## 5. Visualization (Conceptual)
```mermaid
graph TD
subgraph Proposed Pipeline Stages
Start --> Prep[PrepareProcessingItemsStage]
Prep --> ItemLoop{Loop per Item}
ItemLoop -- FileRule --> RegProc[RegularMapProcessorStage]
ItemLoop -- MergeTask --> MergeProc[MergedTaskProcessorStage]
RegProc --> Scale(InitialScalingStage)
MergeProc --> Scale
Scale --> Save[SaveVariantsStage]
Save --> UpdateContext[Update Main Context w/ Results]
UpdateContext --> ItemLoop
end
```
## 6. Benefits
* Improved Readability & Understanding.
* Enhanced Maintainability & Reduced Risk.
* Better Testability.
* Clearer Dependencies.

241
config/app_settings.json
View File

@ -1,245 +1,4 @@
{
"ASSET_TYPE_DEFINITIONS": {
"Surface": {
"description": "A single Standard PBR material set for a surface.",
"color": "#1f3e5d",
"examples": [
"Set: Wood01_COL + Wood01_NRM + WOOD01_ROUGH",
"Set: Dif_Concrete + Normal_Concrete + Refl_Concrete"
]
},
"Model": {
"description": "A set that contains models, can include PBR textureset",
"color": "#b67300",
"examples": [
"Single = Chair.fbx",
"Set = Plant02.fbx + Plant02_col + Plant02_SSS"
]
},
"Decal": {
"description": "A alphamasked textureset",
"color": "#68ac68",
"examples": [
"Set = DecalGraffiti01_Col + DecalGraffiti01_Alpha",
"Single = DecalLeakStain03"
]
},
"Atlas": {
"description": "A texture, name usually hints that it's an atlas",
"color": "#955b8b",
"examples": [
"Set = FoliageAtlas01_col + FoliageAtlas01_nrm"
]
},
"UtilityMap": {
"description": "A useful image-asset consisting of only a single texture. Therefor each Utilitymap can only contain a single item.",
"color": "#706b87",
"examples": [
"Single = imperfection.png",
"Single = smudges.png",
"Single = scratches.tif"
]
}
},
"FILE_TYPE_DEFINITIONS": {
"MAP_COL": {
"description": "Color/Albedo Map",
"color": "#ffaa00",
"examples": [
"_col.",
"_basecolor.",
"albedo",
"diffuse"
],
"standard_type": "COL",
"bit_depth_rule": "force_8bit",
"is_grayscale": false,
"keybind": "C"
},
"MAP_NRM": {
"description": "Normal Map",
"color": "#cca2f1",
"examples": [
"_nrm.",
"_normal."
],
"standard_type": "NRM",
"bit_depth_rule": "respect",
"is_grayscale": false,
"keybind": "N"
},
"MAP_METAL": {
"description": "Metalness Map",
"color": "#dcf4f2",
"examples": [
"_metal.",
"_met."
],
"standard_type": "METAL",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "M"
},
"MAP_ROUGH": {
"description": "Roughness Map",
"color": "#bfd6bf",
"examples": [
"_rough.",
"_rgh.",
"_gloss"
],
"standard_type": "ROUGH",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "R"
},
"MAP_GLOSS": {
"description": "Glossiness Map",
"color": "#d6bfd6",
"examples": [
"_gloss.",
"_gls."
],
"standard_type": "GLOSS",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "R"
},
"MAP_AO": {
"description": "Ambient Occlusion Map",
"color": "#e3c7c7",
"examples": [
"_ao.",
"_ambientocclusion."
],
"standard_type": "AO",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_DISP": {
"description": "Displacement/Height Map",
"color": "#c6ddd5",
"examples": [
"_disp.",
"_height."
],
"standard_type": "DISP",
"bit_depth_rule": "respect",
"is_grayscale": true,
"keybind": "D"
},
"MAP_REFL": {
"description": "Reflection/Specular Map",
"color": "#c2c2b9",
"examples": [
"_refl.",
"_specular."
],
"standard_type": "REFL",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "M"
},
"MAP_SSS": {
"description": "Subsurface Scattering Map",
"color": "#a0d394",
"examples": [
"_sss.",
"_subsurface."
],
"standard_type": "SSS",
"bit_depth_rule": "respect",
"is_grayscale": true
},
"MAP_FUZZ": {
"description": "Fuzz/Sheen Map",
"color": "#a2d1da",
"examples": [
"_fuzz.",
"_sheen."
],
"standard_type": "FUZZ",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_IDMAP": {
"description": "ID Map (for masking)",
"color": "#ca8fb4",
"examples": [
"_id.",
"_matid."
],
"standard_type": "IDMAP",
"bit_depth_rule": "force_8bit",
"is_grayscale": false
},
"MAP_MASK": {
"description": "Generic Mask Map",
"color": "#c6e2bf",
"examples": [
"_mask."
],
"standard_type": "MASK",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_IMPERFECTION": {
"description": "Imperfection Map (scratches, dust)",
"color": "#e6d1a6",
"examples": [
"_imp.",
"_imperfection.",
"splatter",
"scratches",
"smudges",
"hairs",
"fingerprints"
],
"standard_type": "IMPERFECTION",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MODEL": {
"description": "3D Model File",
"color": "#3db2bd",
"examples": [
".fbx",
".obj"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false
},
"EXTRA": {
"description": "asset previews or metadata",
"color": "#8c8c8c",
"examples": [
".txt",
".zip",
"preview.",
"_flat.",
"_sphere.",
"_Cube.",
"thumb"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false,
"keybind": "E"
},
"FILE_IGNORE": {
"description": "File to be ignored",
"color": "#673d35",
"examples": [
"Thumbs.db",
".DS_Store"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false,
"keybind": "X"
}
},
"TARGET_FILENAME_PATTERN": "{base_name}_{map_type}_{resolution}.{ext}",
"RESPECT_VARIANT_MAP_TYPES": [
"COL"

44
config/asset_type_definitions.json Normal file
View File

@ -0,0 +1,44 @@
{
"ASSET_TYPE_DEFINITIONS": {
"Surface": {
"description": "A single Standard PBR material set for a surface.",
"color": "#1f3e5d",
"examples": [
"Set: Wood01_COL + Wood01_NRM + WOOD01_ROUGH",
"Set: Dif_Concrete + Normal_Concrete + Refl_Concrete"
]
},
"Model": {
"description": "A set that contains models, can include PBR textureset",
"color": "#b67300",
"examples": [
"Single = Chair.fbx",
"Set = Plant02.fbx + Plant02_col + Plant02_SSS"
]
},
"Decal": {
"description": "A alphamasked textureset",
"color": "#68ac68",
"examples": [
"Set = DecalGraffiti01_Col + DecalGraffiti01_Alpha",
"Single = DecalLeakStain03"
]
},
"Atlas": {
"description": "A texture, name usually hints that it's an atlas",
"color": "#955b8b",
"examples": [
"Set = FoliageAtlas01_col + FoliageAtlas01_nrm"
]
},
"UtilityMap": {
"description": "A useful image-asset consisting of only a single texture. Therefor each Utilitymap can only contain a single item.",
"color": "#706b87",
"examples": [
"Single = imperfection.png",
"Single = smudges.png",
"Single = scratches.tif"
]
}
}
}

201
config/file_type_definitions.json Normal file
View File

@ -0,0 +1,201 @@
{
"FILE_TYPE_DEFINITIONS": {
"MAP_COL": {
"description": "Color/Albedo Map",
"color": "#ffaa00",
"examples": [
"_col.",
"_basecolor.",
"albedo",
"diffuse"
],
"standard_type": "COL",
"bit_depth_rule": "force_8bit",
"is_grayscale": false,
"keybind": "C"
},
"MAP_NRM": {
"description": "Normal Map",
"color": "#cca2f1",
"examples": [
"_nrm.",
"_normal."
],
"standard_type": "NRM",
"bit_depth_rule": "respect",
"is_grayscale": false,
"keybind": "N"
},
"MAP_METAL": {
"description": "Metalness Map",
"color": "#dcf4f2",
"examples": [
"_metal.",
"_met."
],
"standard_type": "METAL",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "M"
},
"MAP_ROUGH": {
"description": "Roughness Map",
"color": "#bfd6bf",
"examples": [
"_rough.",
"_rgh.",
"_gloss"
],
"standard_type": "ROUGH",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "R"
},
"MAP_GLOSS": {
"description": "Glossiness Map",
"color": "#d6bfd6",
"examples": [
"_gloss.",
"_gls."
],
"standard_type": "GLOSS",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "R"
},
"MAP_AO": {
"description": "Ambient Occlusion Map",
"color": "#e3c7c7",
"examples": [
"_ao.",
"_ambientocclusion."
],
"standard_type": "AO",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_DISP": {
"description": "Displacement/Height Map",
"color": "#c6ddd5",
"examples": [
"_disp.",
"_height."
],
"standard_type": "DISP",
"bit_depth_rule": "respect",
"is_grayscale": true,
"keybind": "D"
},
"MAP_REFL": {
"description": "Reflection/Specular Map",
"color": "#c2c2b9",
"examples": [
"_refl.",
"_specular."
],
"standard_type": "REFL",
"bit_depth_rule": "force_8bit",
"is_grayscale": true,
"keybind": "M"
},
"MAP_SSS": {
"description": "Subsurface Scattering Map",
"color": "#a0d394",
"examples": [
"_sss.",
"_subsurface."
],
"standard_type": "SSS",
"bit_depth_rule": "respect",
"is_grayscale": true
},
"MAP_FUZZ": {
"description": "Fuzz/Sheen Map",
"color": "#a2d1da",
"examples": [
"_fuzz.",
"_sheen."
],
"standard_type": "FUZZ",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_IDMAP": {
"description": "ID Map (for masking)",
"color": "#ca8fb4",
"examples": [
"_id.",
"_matid."
],
"standard_type": "IDMAP",
"bit_depth_rule": "force_8bit",
"is_grayscale": false
},
"MAP_MASK": {
"description": "Generic Mask Map",
"color": "#c6e2bf",
"examples": [
"_mask."
],
"standard_type": "MASK",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MAP_IMPERFECTION": {
"description": "Imperfection Map (scratches, dust)",
"color": "#e6d1a6",
"examples": [
"_imp.",
"_imperfection.",
"splatter",
"scratches",
"smudges",
"hairs",
"fingerprints"
],
"standard_type": "IMPERFECTION",
"bit_depth_rule": "force_8bit",
"is_grayscale": true
},
"MODEL": {
"description": "3D Model File",
"color": "#3db2bd",
"examples": [
".fbx",
".obj"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false
},
"EXTRA": {
"description": "asset previews or metadata",
"color": "#8c8c8c",
"examples": [
".txt",
".zip",
"preview.",
"_flat.",
"_sphere.",
"_Cube.",
"thumb"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false,
"keybind": "E"
},
"FILE_IGNORE": {
"description": "File to be ignored",
"color": "#673d35",
"examples": [
"Thumbs.db",
".DS_Store"
],
"standard_type": "",
"bit_depth_rule": "",
"is_grayscale": false,
"keybind": "X"
}
}
}

0
config/user_settings.json Normal file
View File

243
configuration.py
View File

@ -3,12 +3,16 @@ import os
from pathlib import Path
import logging
import re
import collections.abc
log = logging.getLogger(__name__)
BASE_DIR = Path(__file__).parent
APP_SETTINGS_PATH = BASE_DIR / "config" / "app_settings.json"
LLM_SETTINGS_PATH = BASE_DIR / "config" / "llm_settings.json"
ASSET_TYPE_DEFINITIONS_PATH = BASE_DIR / "config" / "asset_type_definitions.json"
FILE_TYPE_DEFINITIONS_PATH = BASE_DIR / "config" / "file_type_definitions.json"
USER_SETTINGS_PATH = BASE_DIR / "config" / "user_settings.json" # New path for user settings
PRESETS_DIR = BASE_DIR / "Presets"
class ConfigurationError(Exception):
@ -64,6 +68,25 @@ def _fnmatch_to_regex(pattern: str) -> str:
# For filename matching, we usually want to find the pattern, not match the whole string.
return res
def _deep_merge_dicts(base_dict: dict, override_dict: dict) -> dict:
"""
Recursively merges override_dict into base_dict.
If a key exists in both and both values are dicts, it recursively merges them.
Otherwise, the value from override_dict takes precedence.
Modifies base_dict in place and returns it.
"""
for key, value in override_dict.items():
if isinstance(value, collections.abc.Mapping):
node = base_dict.get(key) # Use .get() to avoid creating empty dicts if not needed for override
if isinstance(node, collections.abc.Mapping):
_deep_merge_dicts(node, value) # node is base_dict[key], modified in place
else:
# If base_dict[key] is not a dict or doesn't exist, override it
base_dict[key] = value
else:
base_dict[key] = value
return base_dict
class Configuration:
"""
@ -71,7 +94,7 @@ class Configuration:
"""
def __init__(self, preset_name: str):
"""
Loads core config and the specified preset file.
Loads core config, user overrides, and the specified preset file.
Args:
preset_name: The name of the preset (without .json extension).
@ -81,9 +104,32 @@ class Configuration:
"""
log.debug(f"Initializing Configuration with preset: '{preset_name}'")
self.preset_name = preset_name
# 1. Load core settings
self._core_settings: dict = self._load_core_config()
# 2. Load asset type definitions
self._asset_type_definitions: dict = self._load_asset_type_definitions()
# 3. Load file type definitions
self._file_type_definitions: dict = self._load_file_type_definitions()
# 4. Load user settings
user_settings_overrides: dict = self._load_user_settings()
# 5. Deep merge user settings onto core settings
if user_settings_overrides:
log.info("Applying user setting overrides to core settings.")
# _deep_merge_dicts modifies self._core_settings in place
_deep_merge_dicts(self._core_settings, user_settings_overrides)
# 6. Load LLM settings
self._llm_settings: dict = self._load_llm_config()
# 7. Load preset settings (conceptually overrides combined base + user for shared keys)
self._preset_settings: dict = self._load_preset(preset_name)
# 8. Validate and compile (after all base/user/preset settings are established)
self._validate_configs()
self._compile_regex_patterns()
log.info(f"Configuration loaded successfully using preset: '{self.preset_name}'")
@ -215,9 +261,79 @@ class Configuration:
except Exception as e:
raise ConfigurationError(f"Failed to read preset file {preset_file}: {e}")
def _load_asset_type_definitions(self) -> dict:
"""Loads asset type definitions from the asset_type_definitions.json file."""
log.debug(f"Loading asset type definitions from: {ASSET_TYPE_DEFINITIONS_PATH}")
if not ASSET_TYPE_DEFINITIONS_PATH.is_file():
raise ConfigurationError(f"Asset type definitions file not found: {ASSET_TYPE_DEFINITIONS_PATH}")
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "ASSET_TYPE_DEFINITIONS" not in data:
raise ConfigurationError(f"Key 'ASSET_TYPE_DEFINITIONS' not found in {ASSET_TYPE_DEFINITIONS_PATH}")
settings = data["ASSET_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
raise ConfigurationError(f"'ASSET_TYPE_DEFINITIONS' in {ASSET_TYPE_DEFINITIONS_PATH} must be a dictionary.")
log.debug(f"Asset type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}")
def _load_file_type_definitions(self) -> dict:
"""Loads file type definitions from the file_type_definitions.json file."""
log.debug(f"Loading file type definitions from: {FILE_TYPE_DEFINITIONS_PATH}")
if not FILE_TYPE_DEFINITIONS_PATH.is_file():
raise ConfigurationError(f"File type definitions file not found: {FILE_TYPE_DEFINITIONS_PATH}")
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "FILE_TYPE_DEFINITIONS" not in data:
raise ConfigurationError(f"Key 'FILE_TYPE_DEFINITIONS' not found in {FILE_TYPE_DEFINITIONS_PATH}")
settings = data["FILE_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
raise ConfigurationError(f"'FILE_TYPE_DEFINITIONS' in {FILE_TYPE_DEFINITIONS_PATH} must be a dictionary.")
log.debug(f"File type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}")
def _load_user_settings(self) -> dict:
"""Loads user override settings from config/user_settings.json."""
log.debug(f"Attempting to load user settings from: {USER_SETTINGS_PATH}")
if not USER_SETTINGS_PATH.is_file():
log.info(f"User settings file not found: {USER_SETTINGS_PATH}. Proceeding without user overrides.")
return {}
try:
with open(USER_SETTINGS_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
log.info(f"User settings loaded successfully from {USER_SETTINGS_PATH}.")
return settings
except json.JSONDecodeError as e:
log.warning(f"Failed to parse user settings file {USER_SETTINGS_PATH}: Invalid JSON - {e}. Using empty user settings.")
return {}
except Exception as e:
log.warning(f"Failed to read user settings file {USER_SETTINGS_PATH}: {e}. Using empty user settings.")
return {}
def _validate_configs(self):
"""Performs basic validation checks on loaded settings."""
log.debug("Validating loaded configurations...")
# Validate new definition files first
if not isinstance(self._asset_type_definitions, dict):
raise ConfigurationError("Asset type definitions were not loaded correctly or are not a dictionary.")
if not self._asset_type_definitions: # Check if empty
raise ConfigurationError("Asset type definitions are empty.")
if not isinstance(self._file_type_definitions, dict):
raise ConfigurationError("File type definitions were not loaded correctly or are not a dictionary.")
if not self._file_type_definitions: # Check if empty
raise ConfigurationError("File type definitions are empty.")
# Preset validation
required_preset_keys = [
"preset_name", "supplier_name", "source_naming", "map_type_mapping",
@ -236,7 +352,7 @@ class Configuration:
if 'target_type' not in rule or not isinstance(rule['target_type'], str):
raise ConfigurationError(f"Preset '{self.preset_name}': Rule at index {index} in 'map_type_mapping' is missing 'target_type' string.")
valid_file_type_keys = self._core_settings.get('FILE_TYPE_DEFINITIONS', {}).keys()
valid_file_type_keys = self._file_type_definitions.keys()
if rule['target_type'] not in valid_file_type_keys:
raise ConfigurationError(
f"Preset '{self.preset_name}': Rule at index {index} in 'map_type_mapping' "
@ -261,7 +377,7 @@ class Configuration:
raise ConfigurationError("Core config 'IMAGE_RESOLUTIONS' must be a dictionary.")
# Validate DEFAULT_ASSET_CATEGORY
valid_asset_type_keys = self._core_settings.get('ASSET_TYPE_DEFINITIONS', {}).keys()
valid_asset_type_keys = self._asset_type_definitions.keys()
default_asset_category_value = self._core_settings.get('DEFAULT_ASSET_CATEGORY')
if not default_asset_category_value:
raise ConfigurationError("Core config 'DEFAULT_ASSET_CATEGORY' is missing.")
@ -423,11 +539,11 @@ class Configuration:
Gets the bit depth rule ('respect', 'force_8bit', 'force_16bit') for a given map type identifier.
The map_type_input can be an FTD key (e.g., "MAP_COL") or a suffixed FTD key (e.g., "MAP_COL-1").
"""
if not self._core_settings or 'FILE_TYPE_DEFINITIONS' not in self._core_settings:
log.warning("FILE_TYPE_DEFINITIONS not found in core settings. Cannot determine bit depth rule.")
if not self._file_type_definitions: # Check if the attribute exists and is not empty
log.warning("File type definitions not loaded. Cannot determine bit depth rule.")
return "respect"
file_type_definitions = self._core_settings['FILE_TYPE_DEFINITIONS']
file_type_definitions = self._file_type_definitions
# 1. Try direct match with map_type_input as FTD key
definition = file_type_definitions.get(map_type_input)
@ -473,8 +589,8 @@ class Configuration:
from FILE_TYPE_DEFINITIONS.
"""
aliases = set()
file_type_definitions = self._core_settings.get('FILE_TYPE_DEFINITIONS', {})
for _key, definition in file_type_definitions.items():
# _file_type_definitions is guaranteed to be a dict by the loader
for _key, definition in self._file_type_definitions.items():
if isinstance(definition, dict):
standard_type = definition.get('standard_type')
if standard_type and isinstance(standard_type, str) and standard_type.strip():
@ -482,16 +598,16 @@ class Configuration:
return sorted(list(aliases))
def get_asset_type_definitions(self) -> dict:
"""Returns the ASSET_TYPE_DEFINITIONS dictionary from core settings."""
return self._core_settings.get('ASSET_TYPE_DEFINITIONS', {})
"""Returns the _asset_type_definitions dictionary."""
return self._asset_type_definitions
def get_asset_type_keys(self) -> list:
"""Returns a list of valid asset type keys from core settings."""
return list(self.get_asset_type_definitions().keys())
def get_file_type_definitions_with_examples(self) -> dict:
"""Returns the FILE_TYPE_DEFINITIONS dictionary (including descriptions and examples) from core settings."""
return self._core_settings.get('FILE_TYPE_DEFINITIONS', {})
"""Returns the _file_type_definitions dictionary (including descriptions and examples)."""
return self._file_type_definitions
def get_file_type_keys(self) -> list:
"""Returns a list of valid file type keys from core settings."""
@ -534,7 +650,7 @@ class Configuration:
@property
def FILE_TYPE_DEFINITIONS(self) -> dict:
return self._core_settings.get('FILE_TYPE_DEFINITIONS', {})
return self._file_type_definitions
@property
def keybind_config(self) -> dict[str, list[str]]:
@ -544,8 +660,8 @@ class Configuration:
Example: {'C': ['MAP_COL'], 'R': ['MAP_ROUGH', 'MAP_GLOSS']}
"""
keybinds = {}
file_type_defs = self._core_settings.get('FILE_TYPE_DEFINITIONS', {})
for ftd_key, ftd_value in file_type_defs.items():
# _file_type_definitions is guaranteed to be a dict by the loader
for ftd_key, ftd_value in self._file_type_definitions.items():
if isinstance(ftd_value, dict) and 'keybind' in ftd_value:
key = ftd_value['keybind']
if key not in keybinds:
@ -561,25 +677,92 @@ class Configuration:
def load_base_config() -> dict:
"""
Loads only the base configuration from app_settings.json.
Does not load presets or perform merging/validation.
Loads base configuration by merging app_settings.json, user_settings.json (if exists),
asset_type_definitions.json, and file_type_definitions.json.
Does not load presets or perform full validation beyond basic file loading.
Returns a dictionary containing the merged settings. If app_settings.json
fails to load, an empty dictionary is returned. If other files
fail, errors are logged, and the function proceeds with what has been loaded.
"""
base_settings = {}
# 1. Load app_settings.json (critical)
if not APP_SETTINGS_PATH.is_file():
log.error(f"Base configuration file not found: {APP_SETTINGS_PATH}")
# Return empty dict or raise a specific error if preferred
# For now, return empty dict to allow GUI to potentially start with defaults
log.error(f"Critical: Base application settings file not found: {APP_SETTINGS_PATH}. Returning empty configuration.")
return {}
try:
with open(APP_SETTINGS_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
return settings
base_settings = json.load(f)
log.info(f"Successfully loaded base application settings from: {APP_SETTINGS_PATH}")
except json.JSONDecodeError as e:
log.error(f"Failed to parse base configuration file {APP_SETTINGS_PATH}: Invalid JSON - {e}")
log.error(f"Critical: Failed to parse base application settings file {APP_SETTINGS_PATH}: Invalid JSON - {e}. Returning empty configuration.")
return {}
except Exception as e:
log.error(f"Failed to read base configuration file {APP_SETTINGS_PATH}: {e}")
log.error(f"Critical: Failed to read base application settings file {APP_SETTINGS_PATH}: {e}. Returning empty configuration.")
return {}
# 2. Attempt to load user_settings.json
user_settings_overrides = {}
if USER_SETTINGS_PATH.is_file():
try:
with open(USER_SETTINGS_PATH, 'r', encoding='utf-8') as f:
user_settings_overrides = json.load(f)
log.info(f"User settings loaded successfully for base_config from {USER_SETTINGS_PATH}.")
except json.JSONDecodeError as e:
log.warning(f"Failed to parse user settings file {USER_SETTINGS_PATH} for base_config: Invalid JSON - {e}. Proceeding without these user overrides.")
except Exception as e:
log.warning(f"Failed to read user settings file {USER_SETTINGS_PATH} for base_config: {e}. Proceeding without these user overrides.")
# 3. Deep merge user settings onto base_settings
if user_settings_overrides:
log.info("Applying user setting overrides to base_settings in load_base_config.")
# _deep_merge_dicts modifies base_settings in place
_deep_merge_dicts(base_settings, user_settings_overrides)
# 4. Load asset_type_definitions.json (non-critical, merge if successful)
if not ASSET_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"Asset type definitions file not found: {ASSET_TYPE_DEFINITIONS_PATH}. Proceeding without it.")
else:
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
asset_defs_data = json.load(f)
if "ASSET_TYPE_DEFINITIONS" in asset_defs_data:
if isinstance(asset_defs_data["ASSET_TYPE_DEFINITIONS"], dict):
# Merge into base_settings, which might already contain user overrides
base_settings['ASSET_TYPE_DEFINITIONS'] = asset_defs_data["ASSET_TYPE_DEFINITIONS"]
log.info(f"Successfully loaded and merged ASSET_TYPE_DEFINITIONS from: {ASSET_TYPE_DEFINITIONS_PATH}")
else:
log.error(f"Value under 'ASSET_TYPE_DEFINITIONS' in {ASSET_TYPE_DEFINITIONS_PATH} is not a dictionary. Skipping merge.")
else:
log.error(f"Key 'ASSET_TYPE_DEFINITIONS' not found in {ASSET_TYPE_DEFINITIONS_PATH}. Skipping merge.")
except json.JSONDecodeError as e:
log.error(f"Failed to parse asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}. Skipping merge.")
except Exception as e:
log.error(f"Failed to read asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}. Skipping merge.")
# 5. Load file_type_definitions.json (non-critical, merge if successful)
if not FILE_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"File type definitions file not found: {FILE_TYPE_DEFINITIONS_PATH}. Proceeding without it.")
else:
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
file_defs_data = json.load(f)
if "FILE_TYPE_DEFINITIONS" in file_defs_data:
if isinstance(file_defs_data["FILE_TYPE_DEFINITIONS"], dict):
# Merge into base_settings
base_settings['FILE_TYPE_DEFINITIONS'] = file_defs_data["FILE_TYPE_DEFINITIONS"]
log.info(f"Successfully loaded and merged FILE_TYPE_DEFINITIONS from: {FILE_TYPE_DEFINITIONS_PATH}")
else:
log.error(f"Value under 'FILE_TYPE_DEFINITIONS' in {FILE_TYPE_DEFINITIONS_PATH} is not a dictionary. Skipping merge.")
else:
log.error(f"Key 'FILE_TYPE_DEFINITIONS' not found in {FILE_TYPE_DEFINITIONS_PATH}. Skipping merge.")
except json.JSONDecodeError as e:
log.error(f"Failed to parse file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}. Skipping merge.")
except Exception as e:
log.error(f"Failed to read file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}. Skipping merge.")
return base_settings
def save_llm_config(settings_dict: dict):
"""
Saves the provided LLM settings dictionary to llm_settings.json.
@ -594,6 +777,18 @@ def save_llm_config(settings_dict: dict):
log.error(f"Failed to save LLM configuration file {LLM_SETTINGS_PATH}: {e}")
# Re-raise as ConfigurationError to signal failure upstream
raise ConfigurationError(f"Failed to save LLM configuration: {e}")
def save_user_config(settings_dict: dict):
"""Saves the provided settings dictionary to user_settings.json."""
log.debug(f"Saving user config to: {USER_SETTINGS_PATH}")
try:
# Ensure parent directory exists (though 'config/' should always exist)
USER_SETTINGS_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(USER_SETTINGS_PATH, 'w', encoding='utf-8') as f:
json.dump(settings_dict, f, indent=4)
log.info(f"User config saved successfully to {USER_SETTINGS_PATH}")
except Exception as e:
log.error(f"Failed to save user configuration file {USER_SETTINGS_PATH}: {e}")
raise ConfigurationError(f"Failed to save user configuration: {e}")
def save_base_config(settings_dict: dict):
"""
Saves the provided settings dictionary to app_settings.json.

113
documentation/preferences_refactor_plan.md Normal file
View File

@ -0,0 +1,113 @@
# Refactoring Plan for Preferences Window (ConfigEditorDialog)
## 1. Overview
This document outlines the plan to refactor the preferences window (`gui/config_editor_dialog.py`). The primary goal is to address issues related to misaligned scope, poor user experience for certain data types, and incomplete interactivity. The refactoring will focus on making the `ConfigEditorDialog` a robust editor for settings in `config/app_settings.json` that are intended to be overridden by the user via `config/user_settings.json`.
## 2. Assessment Summary
* **Misaligned Scope:** The dialog currently includes UI for "Asset Type Definitions" and "File Type Definitions". However, these are managed in separate dedicated JSON files ([`config/asset_type_definitions.json`](config/asset_type_definitions.json) and [`config/file_type_definitions.json`](config/file_type_definitions.json)) and are not saved by this dialog (which targets `config/user_settings.json`).
* **Poor UX for Data Types:**
* Lists (e.g., `RESPECT_VARIANT_MAP_TYPES`) are edited as comma-separated strings.
* Dictionary-like structures (e.g., `IMAGE_RESOLUTIONS`) are handled inconsistently (JSON defines as dict, UI attempts list-of-pairs).
* Editing complex list-of-objects (e.g., `MAP_MERGE_RULES`) is functionally incomplete.
* **Incomplete Interactivity:** Many table-based editors lack "Add/Remove Row" functionality and proper cell delegates for intuitive editing.
* **LLM Settings:** Confirmed to be correctly managed by the separate `LLMEditorWidget` and `config/llm_settings.json`, so they are out of scope for this specific dialog refactor.
## 3. Refactoring Phases and Plan Details
```mermaid
graph TD
A[Start: Current State] --> B{Phase 1: Correct Scope & Critical UX/Data Fixes};
B --> C{Phase 2: Enhance MAP_MERGE_RULES Editor};
C --> D{Phase 3: General UX & Table Interactivity};
D --> E[End: Refactored Preferences Window];
subgraph "Phase 1: Correct Scope & Critical UX/Data Fixes"
B1[Remove Definitions Editing from ConfigEditorDialog]
B2[Improve List Editing for RESPECT_VARIANT_MAP_TYPES]
B3[Fix IMAGE_RESOLUTIONS Handling (Dictionary)]
B4[Handle Simple Nested Settings (e.g., general_settings)]
end
subgraph "Phase 2: Enhance MAP_MERGE_RULES Editor"
C1[Implement Add/Remove for Merge Rules]
C2[Improve Rule Detail Editing (ComboBoxes, SpinBoxes)]
end
subgraph "Phase 3: General UX & Table Interactivity"
D1[Implement IMAGE_RESOLUTIONS Table Add/Remove Buttons]
D2[Implement Necessary Table Cell Delegates (e.g., for IMAGE_RESOLUTIONS values)]
D3[Review/Refine Tab Layout & Widget Grouping]
end
B --> B1; B --> B2; B --> B3; B --> B4;
C --> C1; C --> C2;
D --> D1; D --> D2; D --> D3;
```
### Phase 1: Correct Scope & Critical UX/Data Fixes (in `gui/config_editor_dialog.py`)
1. **Remove Definitions Editing:**
* **Action:** In `populate_definitions_tab`, remove the inner `QTabWidget` and the code that creates/populates the "Asset Types" and "File Types" tables.
* The `DEFAULT_ASSET_CATEGORY` `QComboBox` (for the setting from `app_settings.json`) should remain. Its items should be populated using keys obtained from the `Configuration` class (which loads the actual `ASSET_TYPE_DEFINITIONS` from its dedicated file).
* **Rationale:** Simplifies the dialog to settings managed via `user_settings.json`. Editing of the full definition files requires dedicated UI (see Future Enhancements note).
2. **Improve `RESPECT_VARIANT_MAP_TYPES` Editing:**
* **Action:** In `populate_output_naming_tab`, replace the `QLineEdit` for `RESPECT_VARIANT_MAP_TYPES` with a `QListWidget` and "Add"/"Remove" buttons.
* "Add" button: Use `QInputDialog.getItem` with items populated from `Configuration.get_file_type_keys()` (or similar method accessing loaded `FILE_TYPE_DEFINITIONS`) to allow users to select a valid file type key.
* "Remove" button: Remove the selected item from the `QListWidget`.
* Update `save_settings` to read the list of strings from this `QListWidget`.
* Update `populate_widgets_from_settings` to populate this `QListWidget`.
3. **Fix `IMAGE_RESOLUTIONS` Handling:**
* **Action:** In `populate_image_processing_tab`:
* The `QTableWidget` for `IMAGE_RESOLUTIONS` should have two columns: "Name" (string, for the dictionary key) and "Resolution (px)" (integer, for the dictionary value).
* In `populate_image_resolutions_table`, ensure it correctly populates from the dictionary structure in `self.settings['IMAGE_RESOLUTIONS']` (from `app_settings.json`).
* In `save_settings`, ensure it correctly reads data from the table and reconstructs the `IMAGE_RESOLUTIONS` dictionary (e.g., `{"4K": 4096, "2K": 2048}`) when saving to `user_settings.json`.
* ComboBoxes `CALCULATE_STATS_RESOLUTION` and `RESOLUTION_THRESHOLD_FOR_JPG` should be populated with the *keys* (names like "4K", "2K") from the `IMAGE_RESOLUTIONS` dictionary. `RESOLUTION_THRESHOLD_FOR_JPG` should also include "Never" and "Always" options. The `save_settings` method needs to correctly map these special ComboBox values back to appropriate storable values if necessary (e.g., sentinel numbers or specific strings if the backend configuration expects them for "Never"/"Always").
4. **Handle Simple Nested Settings (e.g., `general_settings`):**
* **Action:** For `general_settings.invert_normal_map_green_channel_globally` (from `config/app_settings.json`):
* Add a `QCheckBox` labeled "Invert Normal Map Green Channel Globally" to an appropriate tab (e.g., "Image Processing" or a "General" tab after layout review).
* Update `populate_widgets_from_settings` to read `self.settings.get('general_settings', {}).get('invert_normal_map_green_channel_globally', False)`.
* Update `save_settings` to write this value back to `target_file_content.setdefault('general_settings', {})['invert_normal_map_green_channel_globally'] = widget.isChecked()`.
### Phase 2: Enhance `MAP_MERGE_RULES` Editor (in `gui/config_editor_dialog.py`)
1. **Rule Management:**
* **Action:** In `populate_map_merging_tab`:
* Connect the "Add Rule" button:
* Create a default new rule dictionary (e.g., `{"output_map_type": "NEW_RULE", "inputs": {}, "defaults": {}, "output_bit_depth": "respect_inputs"}`).
* Add it to the internal list of rules that will be saved (e.g., a copy of `self.settings['MAP_MERGE_RULES']` that gets modified).
* Add a new `QListWidgetItem` for it and select it to display its details.
* Connect the "Remove Rule" button:
* Remove the selected rule from the internal list and the `QListWidget`.
* Clear the details panel.
2. **Rule Details Panel Improvements (`display_merge_rule_details`):**
* **`output_map_type`:** Change the `QLineEdit` to a `QComboBox`. Populate its items from `Configuration.get_file_type_keys()`.
* **`inputs` Table:** The "Input Map Type" column cells should use a `QComboBox` delegate, populated with `Configuration.get_file_type_keys()` plus an empty/None option.
* **`defaults` Table:** The "Default Value" column cells should use a `QDoubleSpinBox` delegate (e.g., range 0.0 to 1.0, or 0-255 if appropriate for specific channel types).
* Ensure changes in these detail editors update the underlying rule data associated with the selected `QListWidgetItem` and the internal list of rules.
### Phase 3: General UX & Table Interactivity (in `gui/config_editor_dialog.py`)
1. **Implement `IMAGE_RESOLUTIONS` Table Add/Remove Buttons:**
* **Action:** In `populate_image_processing_tab`, connect the "Add Row" and "Remove Row" buttons for the `IMAGE_RESOLUTIONS` table.
* "Add Row": Prompt for "Name" (string) and "Resolution (px)" (integer).
* "Remove Row": Remove the selected row from the table and the underlying data.
2. **Implement Necessary Table Cell Delegates:**
* **Action:** For the `IMAGE_RESOLUTIONS` table, the "Resolution (px)" column should use a `QSpinBox` delegate or a `QLineEdit` with integer validation to ensure correct data input.
3. **Review/Refine Tab Layout & Widget Grouping:**
* **Action:** After the functional changes, review the overall layout of tabs and the grouping of settings within `gui/config_editor_dialog.py`.
* Ensure settings from `config/app_settings.json` are logically placed and clearly labeled.
* Verify widget labels are descriptive and tooltips are helpful where needed.
* Confirm correct mapping between UI widgets and the keys in `app_settings.json` (e.g., `OUTPUT_FILENAME_PATTERN` vs. `TARGET_FILENAME_PATTERN`).
## 4. Future Enhancements (Out of Scope for this Refactor)
* **Dedicated Editors for Definitions:** As per user feedback, if `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` require UI-based editing, dedicated dialogs/widgets should be created. These would read from and save to their respective files ([`config/asset_type_definitions.json`](config/asset_type_definitions.json) and [`config/file_type_definitions.json`](config/file_type_definitions.json)) and could adopt a list/details UI similar to the `MAP_MERGE_RULES` editor.
* **Live Updates:** Consider mechanisms for applying some settings without requiring an application restart, if feasible for specific settings.
This plan aims to create a more focused, usable, and correct preferences window.

305
gui/config_editor_dialog.py
View File

@ -1,5 +1,7 @@
import json
import os # Added for path operations
import copy # Added for deepcopy
from PySide6.QtWidgets import (
QDialog, QVBoxLayout, QHBoxLayout, QTabWidget, QWidget,
QLineEdit, QSpinBox, QDoubleSpinBox, QComboBox, QCheckBox,
@ -14,10 +16,10 @@ from PySide6.QtWidgets import QColorDialog, QStyledItemDelegate, QApplication
# Assuming configuration.py is in the parent directory or accessible
try:
from configuration import load_base_config, save_base_config
from configuration import load_base_config, save_user_config, ConfigurationError
except ImportError:
# Fallback import for testing or different project structure
from ..configuration import load_base_config, save_base_config
from ..configuration import load_base_config, save_user_config, ConfigurationError
# --- Custom Delegate for Color Editing ---
@ -86,10 +88,29 @@ class ConfigEditorDialog(QDialog):
"""Loads settings from the configuration file."""
try:
self.settings = load_base_config()
print("Configuration loaded successfully.") # Debug print
# Store a deep copy of the initial user-configurable settings for granular save.
# These are settings from the effective configuration (base + user + defs)
# that this dialog manages and are intended for user_settings.json.
# Exclude definitions that are stored in separate files or not directly managed here.
self.original_user_configurable_settings = {} # Initialize first
if self.settings: # Ensure settings were loaded
keys_to_copy = [
k for k in self.settings
if k not in ["ASSET_TYPE_DEFINITIONS", "FILE_TYPE_DEFINITIONS"]
]
# Create a temporary dictionary with only the keys to be copied
temp_original_settings = {
k: self.settings[k] for k in keys_to_copy if k in self.settings
}
self.original_user_configurable_settings = copy.deepcopy(temp_original_settings)
print("Original user-configurable settings (relevant parts) deep copied for comparison.") # Debug print
else:
# If self.settings is None or empty, original_user_configurable_settings remains an empty dict.
print("Settings not loaded or empty; original_user_configurable_settings initialized as empty.") # Debug print
except Exception as e:
QMessageBox.critical(self, "Loading Error", f"Failed to load configuration: {e}")
self.settings = {} # Use empty settings on failure
self.original_user_configurable_settings = {}
# Optionally disable save button or widgets if loading fails
self.button_box.button(QDialogButtonBox.Save).setEnabled(False)
@ -851,154 +872,162 @@ class ConfigEditorDialog(QDialog):
widget.setText(color.name()) # Get color as hex string
def save_settings(self):
"""Reads values from widgets and saves them to the configuration file."""
new_settings = {}
"""
Reads values from widgets, compares them to the original loaded settings,
and saves only the changed values to config/user_settings.json, preserving
other existing user settings.
"""
# 1a. Load Current Target File (user_settings.json)
# Assuming configuration.py and this file are structured such that
# 'config/user_settings.json' is the correct relative path from the workspace root.
# TODO: Ideally, get this path from the configuration module if it provides a constant.
user_settings_path = os.path.join("config", "user_settings.json")
# Start with a deep copy of the original settings structure to preserve
# sections/keys that might not have dedicated widgets (though ideally all should)
import copy
new_settings = copy.deepcopy(self.settings)
target_file_content = {}
if os.path.exists(user_settings_path):
try:
with open(user_settings_path, 'r') as f:
target_file_content = json.load(f)
except json.JSONDecodeError:
QMessageBox.warning(self, "Warning",
f"File {user_settings_path} is corrupted or not valid JSON. "
f"It will be overwritten if changes are saved.")
target_file_content = {} # Start fresh if corrupted
except Exception as e:
QMessageBox.critical(self, "Error Loading User Settings",
f"Failed to load {user_settings_path}: {e}. "
f"Proceeding with empty user settings for this save operation.")
target_file_content = {}
# 1b. Get current settings from UI by populating a full settings dictionary
# This `full_ui_state` will mirror the structure of `self.settings` but with current UI values.
full_ui_state = copy.deepcopy(self.settings) # Start with the loaded settings structure
# Iterate through the stored widgets and update the new_settings dictionary
for key, widget in self.widgets.items():
# --- Populate full_ui_state from ALL widgets (adapted from original save_settings logic) ---
# This loop iterates through all widgets and updates `full_ui_state` with their current values.
# It handles simple widgets and complex ones like tables (though table data for definitions
# won't end up in user_settings.json).
for widget_config_key, widget_obj in self.widgets.items():
# `widget_config_key` is the key used in `self.widgets` (e.g., "OUTPUT_BASE_DIR", "IMAGE_RESOLUTIONS_TABLE")
# `keys_path` is for navigating the `full_ui_state` dictionary if `widget_config_key` implies a path.
# For most simple widgets, `widget_config_key` is a direct top-level key.
keys_path = widget_config_key.split('.')
current_level_dict = full_ui_state
# Navigate to the correct dictionary level if widget_config_key is a path like "general.foo"
# For simple keys like "OUTPUT_BASE_DIR", this loop runs once for the key itself.
for i, part_of_key in enumerate(keys_path):
if i == len(keys_path) - 1: # Last part of the key, time to set the value
# Handle simple widgets
if isinstance(widget, (QLineEdit, QSpinBox, QDoubleSpinBox, QCheckBox, QComboBox)):
# Split the key to navigate the dictionary structure
keys = key.split('.')
current_dict = new_settings
for i, k in enumerate(keys):
if i == len(keys) - 1:
# This is the final key, update the value
if isinstance(widget, QLineEdit):
# Handle simple lists displayed as comma-separated strings
if key == "RESPECT_VARIANT_MAP_TYPES":
current_dict[k] = [item.strip() for item in widget.text().split(',') if item.strip()]
if isinstance(widget_obj, QLineEdit):
if widget_config_key == "RESPECT_VARIANT_MAP_TYPES": # Special list handling
current_level_dict[part_of_key] = [item.strip() for item in widget_obj.text().split(',') if item.strip()]
else:
current_dict[k] = widget.text()
elif isinstance(widget, QSpinBox):
current_dict[k] = widget.value()
elif isinstance(widget, QDoubleSpinBox):
current_dict[k] = widget.value()
elif isinstance(widget, QCheckBox):
current_dict[k] = widget.isChecked()
elif isinstance(widget, QComboBox):
# Special handling for RESOLUTION_THRESHOLD_FOR_JPG
if key == "RESOLUTION_THRESHOLD_FOR_JPG": # Use 'key' from the loop, not 'full_key'
selected_text = widget.currentText()
image_resolutions = new_settings.get('IMAGE_RESOLUTIONS', {})
if selected_text == "Never":
# Use a very large number so the comparison target_dim_px >= threshold is always false
current_dict[k] = 999999
elif selected_text == "Always":
current_dict[k] = 1 # Assuming 1 means always apply (any positive dimension >= 1)
elif selected_text in image_resolutions:
current_dict[k] = image_resolutions[selected_text]
current_level_dict[part_of_key] = widget_obj.text()
elif isinstance(widget_obj, QSpinBox):
current_level_dict[part_of_key] = widget_obj.value()
elif isinstance(widget_obj, QDoubleSpinBox):
current_level_dict[part_of_key] = widget_obj.value()
elif isinstance(widget_obj, QCheckBox):
current_level_dict[part_of_key] = widget_obj.isChecked()
elif isinstance(widget_obj, QComboBox):
if widget_config_key == "RESOLUTION_THRESHOLD_FOR_JPG":
selected_text = widget_obj.currentText()
# Use image_resolutions from the potentially modified full_ui_state
image_resolutions_data = full_ui_state.get('IMAGE_RESOLUTIONS', {})
if selected_text == "Never": current_level_dict[part_of_key] = 999999
elif selected_text == "Always": current_level_dict[part_of_key] = 1
elif isinstance(image_resolutions_data, list): # Check if it's the list of [name, val]
found_res = next((res[1] for res in image_resolutions_data if res[0] == selected_text), None)
if found_res is not None: current_level_dict[part_of_key] = found_res
else: current_level_dict[part_of_key] = selected_text # Fallback
elif isinstance(image_resolutions_data, dict) and selected_text in image_resolutions_data: # Original format
current_level_dict[part_of_key] = image_resolutions_data[selected_text]
else: current_level_dict[part_of_key] = selected_text # Fallback
else:
# Fallback or error handling if text is unexpected
print(f"Warning: Unexpected value '{selected_text}' for RESOLUTION_THRESHOLD_FOR_JPG. Saving as text.")
current_dict[k] = selected_text # Save original text as fallback
else:
# Default behavior for other combo boxes
current_dict[k] = widget.currentText()
else:
# Navigate to the next level
# Use full_key for error message consistency
if k not in current_dict or not isinstance(current_dict[k], dict):
# This should not happen if create_tabs is correct, but handle defensively
print(f"Warning: Structure mismatch for key part '{k}' in '{key}'")
break # Stop processing this key
current_dict = current_dict[k]
# Handle TableWidgets (for definitions and image resolutions)
elif key == "ASSET_TYPE_DEFINITIONS_TABLE":
table = widget
asset_defs = {}
current_level_dict[part_of_key] = widget_obj.currentText()
# Handle TableWidgets - only for those relevant to user_settings.json
# ASSET_TYPE_DEFINITIONS and FILE_TYPE_DEFINITIONS are handled by their own files.
# IMAGE_RESOLUTIONS and MAP_MERGE_RULES might be in user_settings.json.
elif widget_config_key == "IMAGE_RESOLUTIONS_TABLE" and isinstance(widget_obj, QTableWidget):
table = widget_obj
resolutions_list = []
for row in range(table.rowCount()):
name_item = table.item(row, 0)
res_item = table.item(row, 1)
if name_item and name_item.text() and res_item and res_item.text():
try:
type_name_item = table.item(row, 0)
desc_item = table.item(row, 1)
color_item = table.item(row, 2) # Delegate stores color in EditRole
examples_item = table.item(row, 3)
if not type_name_item or not type_name_item.text():
print(f"Warning: Skipping row {row} in {key} due to missing type name.")
continue
type_name = type_name_item.text()
description = desc_item.text() if desc_item else ""
# Get color from EditRole data set by the delegate
color_str = color_item.data(Qt.EditRole) if color_item else "#ffffff"
examples_str = examples_item.text() if examples_item else ""
examples_list = [ex.strip() for ex in examples_str.split(',') if ex.strip()]
asset_defs[type_name] = {
"description": description,
"color": color_str,
"examples": examples_list
}
# Assuming resolution value might be int or string like "1024" or "1k"
# For simplicity, store as string if not easily int.
# The config system should handle parsing later.
res_val_str = res_item.text()
try:
res_value = int(res_val_str)
except ValueError:
res_value = res_val_str # Keep as string if not simple int
resolutions_list.append([name_item.text(), res_value])
except Exception as e:
print(f"Error processing row {row} in {key}: {e}")
new_settings['ASSET_TYPE_DEFINITIONS'] = asset_defs # Overwrite with new data
print(f"Skipping resolution row {row} due to error: {e}")
full_ui_state['IMAGE_RESOLUTIONS'] = resolutions_list # Key in settings is IMAGE_RESOLUTIONS
elif key == "FILE_TYPE_DEFINITIONS_TABLE":
table = widget
file_defs = {}
for row in range(table.rowCount()):
# MAP_MERGE_RULES are complex; current save logic is a pass-through.
# For granular save, if MAP_MERGE_RULES are edited, the full updated list should be in full_ui_state.
# The original code had a placeholder for MAP_MERGE_RULES_DATA.
# Assuming if MAP_MERGE_RULES are part of user_settings, their full structure from UI
# would be placed into full_ui_state['MAP_MERGE_RULES'].
# The current implementation of populate_map_merging_tab and display_merge_rule_details
# would need to ensure that `full_ui_state['MAP_MERGE_RULES']` is correctly updated
# if changes are made via the UI. The original save logic for this was:
# `elif key == "MAP_MERGE_RULES_DATA": pass`
# This means `full_ui_state['MAP_MERGE_RULES']` would retain its original loaded value unless
# other UI interactions (like Add/Remove Rule buttons, if implemented and connected) modify it.
# For now, we assume `full_ui_state['MAP_MERGE_RULES']` reflects the intended UI state.
else: # Navigate deeper
if part_of_key not in current_level_dict or not isinstance(current_level_dict[part_of_key], dict):
# This case should ideally not happen if widget keys match settings structure
# or if the base `full_ui_state` (from `self.settings`) had the correct structure.
# If a path implies a dict that doesn't exist, create it.
current_level_dict[part_of_key] = {}
current_level_dict = current_level_dict[part_of_key]
# --- End of populating full_ui_state ---
# 2. Identify Changes by comparing with self.original_user_configurable_settings
changed_settings_count = 0
for key_to_check, original_value in self.original_user_configurable_settings.items():
# `key_to_check` is a top-level key from the user-configurable settings
# (e.g., "OUTPUT_BASE_DIR", "PNG_COMPRESSION_LEVEL", "IMAGE_RESOLUTIONS").
current_value_from_ui = full_ui_state.get(key_to_check)
# Perform comparison. Python's default `!=` works for deep comparison
# of basic types, lists, and dicts if order doesn't matter for lists
# where it shouldn't (e.g. list of strings) or if dicts are canonical.
if current_value_from_ui != original_value:
# This setting has changed. Update it in target_file_content.
# This replaces the whole value for `key_to_check` in user_settings.
target_file_content[key_to_check] = copy.deepcopy(current_value_from_ui)
changed_settings_count += 1
print(f"Setting '{key_to_check}' changed. Old: {original_value}, New: {current_value_from_ui}")
# 3. Save Updated Content to user_settings.json
if changed_settings_count > 0 or not os.path.exists(user_settings_path):
# Save if there are changes or if the file didn't exist (to create it with defaults if any were set)
try:
type_id_item = table.item(row, 0)
desc_item = table.item(row, 1)
color_item = table.item(row, 2) # Delegate stores color in EditRole
examples_item = table.item(row, 3)
std_type_item = table.item(row, 4)
bit_depth_item = table.item(row, 5)
if not type_id_item or not type_id_item.text():
print(f"Warning: Skipping row {row} in {key} due to missing type ID.")
continue
type_id = type_id_item.text()
description = desc_item.text() if desc_item else ""
# Get color from EditRole data set by the delegate
color_str = color_item.data(Qt.EditRole) if color_item else "#ffffff"
examples_str = examples_item.text() if examples_item else ""
examples_list = [ex.strip() for ex in examples_str.split(',') if ex.strip()]
standard_type = std_type_item.text() if std_type_item else ""
bit_depth_rule = bit_depth_item.text() if bit_depth_item else "respect" # Default?
file_defs[type_id] = {
"description": description,
"color": color_str,
"examples": examples_list,
"standard_type": standard_type,
"bit_depth_rule": bit_depth_rule
}
except Exception as e:
print(f"Error processing row {row} in {key}: {e}")
new_settings['FILE_TYPE_DEFINITIONS'] = file_defs # Overwrite with new data
elif key in ["IMAGE_RESOLUTIONS_TABLE", "MAP_BIT_DEPTH_RULES_TABLE"]:
# Still not implemented for these tables
print(f"Warning: Saving for table widget '{key}' is not fully implemented.")
pass # Defer saving complex table data
# Handle Map Merge Rules (more complex)
# Note: Changes made in the details form are NOT saved with this implementation
# due to deferred complexity in updating the list item's data and reconstructing the list.
elif key == "MAP_MERGE_RULES_DATA":
# The original data is stored, but changes in the details form are not reflected.
# A full implementation would require updating the rule data in the list item
# when the details form is edited and then reconstructing the list.
print(f"Warning: Saving for Map Merge Rules details is not fully implemented.")
pass # Defer saving changes from the details form
# Save the new settings
try:
save_base_config(new_settings)
QMessageBox.information(self, "Settings Saved", "Configuration saved successfully.\nRestart the application to apply changes.")
save_user_config(target_file_content) # save_user_config is imported
QMessageBox.information(self, "Settings Saved",
f"User settings saved successfully to {user_settings_path}.\n"
f"{changed_settings_count} setting(s) updated. "
"Some changes may require an application restart.")
self.accept() # Close the dialog
except ConfigurationError as e:
QMessageBox.critical(self, "Saving Error", f"Failed to save user configuration: {e}")
except Exception as e:
QMessageBox.critical(self, "Saving Error", f"Failed to save configuration: {e}")
QMessageBox.critical(self, "Saving Error", f"An unexpected error occurred while saving: {e}")
else:
QMessageBox.information(self, "No Changes", "No changes were made to user-configurable settings.")
self.accept() # Close the dialog, or self.reject() if no changes means cancel
def populate_widgets_from_settings(self):
"""Populates the created widgets with loaded settings."""

97
gui/llm_editor_widget.py
View File

@ -1,6 +1,7 @@
# gui/llm_editor_widget.py
import json
import logging
import copy # Added for deepcopy
from PySide6.QtWidgets import (
QWidget, QVBoxLayout, QTabWidget, QPlainTextEdit, QGroupBox,
QHBoxLayout, QPushButton, QFormLayout, QLineEdit, QDoubleSpinBox,
@ -24,6 +25,7 @@ class LLMEditorWidget(QWidget):
def __init__(self, parent=None):
super().__init__(parent)
self._unsaved_changes = False
self.original_llm_settings = {} # Initialize original_llm_settings
self._init_ui()
self._connect_signals()
self.save_button.setEnabled(False) # Initially disabled
@ -131,6 +133,7 @@ class LLMEditorWidget(QWidget):
try:
with open(LLM_CONFIG_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
self.original_llm_settings = copy.deepcopy(settings) # Store a deep copy
# Populate Prompt Settings
self.prompt_editor.setPlainText(settings.get("llm_predictor_prompt", ""))
@ -159,9 +162,9 @@ class LLMEditorWidget(QWidget):
logger.info("LLM settings loaded successfully.")
except FileNotFoundError:
logger.warning(f"LLM settings file not found: {LLM_CONFIG_PATH}. Using defaults and disabling editor.")
logger.warning(f"LLM settings file not found: {LLM_CONFIG_PATH}. Using defaults.")
QMessageBox.warning(self, "Load Error",
f"LLM settings file not found:\n{LLM_CONFIG_PATH}\n\nPlease ensure the file exists. Using default values.")
f"LLM settings file not found:\n{LLM_CONFIG_PATH}\n\nNew settings will be created if you save.")
# Reset to defaults (optional, or leave fields empty)
self.prompt_editor.clear()
self.endpoint_url_edit.clear()
@ -169,19 +172,21 @@ class LLMEditorWidget(QWidget):
self.model_name_edit.clear()
self.temperature_spinbox.setValue(0.7)
self.timeout_spinbox.setValue(120)
# self.setEnabled(False) # Disabling might be too harsh if user wants to create settings
self.original_llm_settings = {} # Start with empty original settings if file not found
except json.JSONDecodeError as e:
logger.error(f"Error decoding JSON from {LLM_CONFIG_PATH}: {e}")
QMessageBox.critical(self, "Load Error",
f"Failed to parse LLM settings file:\n{LLM_CONFIG_PATH}\n\nError: {e}\n\nPlease check the file for syntax errors. Editor will be disabled.")
self.setEnabled(False) # Disable editor on critical load error
self.original_llm_settings = {} # Reset original settings on JSON error
except Exception as e: # Catch other potential errors during loading/populating
logger.error(f"An unexpected error occurred loading LLM settings: {e}", exc_info=True)
QMessageBox.critical(self, "Load Error",
f"An unexpected error occurred while loading settings:\n{e}\n\nEditor will be disabled.")
self.setEnabled(False)
self.original_llm_settings = {} # Reset original settings on other errors
# Reset unsaved changes flag and disable save button after loading
@ -201,26 +206,38 @@ class LLMEditorWidget(QWidget):
"""Gather data from UI, save to JSON file, and handle errors."""
logger.info("Attempting to save LLM settings...")
settings_dict = {}
# 1.a. Load Current Target File
target_file_content = {}
try:
with open(LLM_CONFIG_PATH, 'r', encoding='utf-8') as f:
target_file_content = json.load(f)
except FileNotFoundError:
logger.info(f"{LLM_CONFIG_PATH} not found. Will create a new one.")
target_file_content = {} # Start with an empty dict if file doesn't exist
except json.JSONDecodeError as e:
logger.error(f"Error decoding existing {LLM_CONFIG_PATH}: {e}. Starting with an empty config for save.")
QMessageBox.warning(self, "Warning",
f"Could not parse existing LLM settings file ({LLM_CONFIG_PATH}).\n"
f"Any pre-existing settings in that file might be overwritten if you save now.\nError: {e}")
target_file_content = {} # Start fresh if current file is corrupt
# 1.b. Gather current UI settings into current_llm_settings
current_llm_settings = {}
parsed_examples = []
has_errors = False
has_errors = False # For example parsing
# Gather API Settings
settings_dict["llm_endpoint_url"] = self.endpoint_url_edit.text().strip()
settings_dict["llm_api_key"] = self.api_key_edit.text() # Keep as is, don't strip
settings_dict["llm_model_name"] = self.model_name_edit.text().strip()
settings_dict["llm_temperature"] = self.temperature_spinbox.value()
settings_dict["llm_request_timeout"] = self.timeout_spinbox.value()
current_llm_settings["llm_endpoint_url"] = self.endpoint_url_edit.text().strip()
current_llm_settings["llm_api_key"] = self.api_key_edit.text() # Keep as is
current_llm_settings["llm_model_name"] = self.model_name_edit.text().strip()
current_llm_settings["llm_temperature"] = self.temperature_spinbox.value()
current_llm_settings["llm_request_timeout"] = self.timeout_spinbox.value()
current_llm_settings["llm_predictor_prompt"] = self.prompt_editor.toPlainText().strip()
# Gather Prompt Settings
settings_dict["llm_predictor_prompt"] = self.prompt_editor.toPlainText().strip()
# Gather and Parse Examples
for i in range(self.examples_tab_widget.count()):
example_editor = self.examples_tab_widget.widget(i)
if isinstance(example_editor, QTextEdit):
example_text = example_editor.toPlainText().strip()
if not example_text: # Skip empty examples silently
if not example_text:
continue
try:
parsed_example = json.loads(example_text)
@ -231,40 +248,58 @@ class LLMEditorWidget(QWidget):
logger.warning(f"Invalid JSON in '{tab_name}': {e}. Skipping example.")
QMessageBox.warning(self, "Invalid Example",
f"The content in '{tab_name}' is not valid JSON and will not be saved.\n\nError: {e}\n\nPlease correct it or remove the tab.")
# Optionally switch to the tab with the error:
# self.examples_tab_widget.setCurrentIndex(i)
else:
logger.warning(f"Widget at index {i} in examples tab is not a QTextEdit. Skipping.")
if has_errors:
logger.warning("LLM settings not saved due to invalid JSON in examples.")
# Keep save button enabled if there were errors, allowing user to fix and retry
# self.save_button.setEnabled(True)
# self._unsaved_changes = True
return # Stop saving process
return
settings_dict["llm_predictor_examples"] = parsed_examples
current_llm_settings["llm_predictor_examples"] = parsed_examples
# Save the dictionary to file
# 1.c. Identify Changes and Update Target File Content
changed_settings_count = 0
for key, current_value in current_llm_settings.items():
original_value = self.original_llm_settings.get(key)
# Special handling for lists (e.g., examples) - direct comparison works
# For other types, direct comparison also works.
# This includes new keys present in current_llm_settings but not in original_llm_settings
if key not in self.original_llm_settings or current_value != original_value:
target_file_content[key] = current_value
logger.debug(f"Setting '{key}' changed or added. Old: '{original_value}', New: '{current_value}'")
changed_settings_count +=1
if changed_settings_count == 0 and self._unsaved_changes:
logger.info("Save called, but no actual changes detected compared to original loaded settings.")
# If _unsaved_changes was true, it means UI interaction happened,
# but values might have been reverted to original.
# We still proceed to save target_file_content as it might contain
# values from a file that was modified externally since last load.
# Or, if the file didn't exist, it will now be created with current UI values.
# 1.d. Save Updated Content
try:
save_llm_config(settings_dict)
save_llm_config(target_file_content) # Save the potentially modified target_file_content
QMessageBox.information(self, "Save Successful", f"LLM settings saved to:\n{LLM_CONFIG_PATH}")
# Update original_llm_settings to reflect the newly saved state
self.original_llm_settings = copy.deepcopy(target_file_content)
self.save_button.setEnabled(False)
self._unsaved_changes = False
self.settings_saved.emit() # Notify MainWindow or others
self.settings_saved.emit()
logger.info("LLM settings saved successfully.")
except ConfigurationError as e:
logger.error(f"Failed to save LLM settings: {e}")
QMessageBox.critical(self, "Save Error", f"Could not save LLM settings.\n\nError: {e}")
# Keep save button enabled as save failed
self.save_button.setEnabled(True)
self.save_button.setEnabled(True) # Keep save enabled
self._unsaved_changes = True
except Exception as e: # Catch unexpected errors during save
except Exception as e:
logger.error(f"An unexpected error occurred during LLM settings save: {e}", exc_info=True)
QMessageBox.critical(self, "Save Error", f"An unexpected error occurred while saving settings:\n{e}")
self.save_button.setEnabled(True)
self.save_button.setEnabled(True) # Keep save enabled
self._unsaved_changes = True
# --- Example Management Slots ---