Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Major Terminogy unification and refactor [Needs thorough testing]

Browse Source
This commit is contained in:
Rusfort
2025-05-06 18:26:26 +02:00
parent 0a3100d448
commit ff548e902e
20 changed files with 689 additions and 356 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
Documentation/01_User_Guide/04_Configuration_and_Presets.md
View File

@@ -9,7 +9,7 @@ The tool's core settings are now stored in `config/app_settings.json`. This JSON
The `configuration.py` module is responsible for loading the settings from `app_settings.json` (including loading and saving the JSON content), merging them with the rules from the selected preset file, and providing the base configuration via the `load_base_config()` function. Note that the old `config.py` file has been deleted.
The `app_settings.json` file is structured into several key sections, including:
* `FILE_TYPE_DEFINITIONS`: Defines known file types (like different texture maps, models, etc.) and their properties. Each definition now includes a `"standard_type"` key for aliasing to a common type and a `"bit_depth_rule"` key specifying how to handle bit depth for this file type. The separate `MAP_BIT_DEPTH_RULES` section has been removed.
* `FILE_TYPE_DEFINITIONS`: Defines known file types (like different texture maps, models, etc.) and their properties. Each definition now includes a `"standard_type"` key for aliasing to a common type (e.g., "COL" for color maps, "NRM" for normal maps), an `"is_grayscale"` boolean property, and a `"bit_depth_rule"` key specifying how to handle bit depth for this file type. The separate `MAP_BIT_DEPTH_RULES` section has been removed. For users creating or editing presets, it's important to note that internal mapping rules (like `Map_type_Mapping.target_type` within a preset's `FileRule`) now directly use the main keys from these `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_RGH"`), not just the `standard_type` aliases.
* `ASSET_TYPE_DEFINITIONS`: Defines known asset types (like Surface, Model, Decal) and their properties.
* `MAP_MERGE_RULES`: Defines how multiple input maps can be merged into a single output map (e.g., combining Normal and Roughness into one).
@@ -40,7 +40,7 @@ Preset files define supplier-specific rules for interpreting asset source files.
* Presets are located in the `presets/` directory.
* Each preset is a JSON file named after the supplier (e.g., `Poliigon.json`).
* Presets contain rules based on filename patterns and keywords to identify map types, models, and other files.
* They also define how variants (like different resolutions or bit depths) are handled and how asset names and categories are determined from the source filename.
* They also define how variants (like different resolutions or bit depths) are handled and how asset names and categories are determined from the source filename. When defining `map_type_mapping` rules within a preset, the `target_type` field must now use a valid key from the `FILE_TYPE_DEFINITIONS` in `config/app_settings.json` (e.g., `"MAP_AO"` instead of a custom alias like `"AO"`).
When processing assets, you must specify which preset to use. The tool then loads the core settings from `config/app_settings.json` and merges them with the rules from the selected preset to determine how to process the input.

2
Documentation/01_User_Guide/09_Output_Structure.md
View File

@@ -23,7 +23,7 @@ The following tokens can be used in both `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_
* `[Time]`: Current time (`HHMMSS`).
* `[Sha5]`: The first 5 characters of the SHA-256 hash of the original input source file (e.g., the source zip archive).
* `[ApplicationPath]`: Absolute path to the application directory.
* `[maptype]`: Specific map type (e.g., `Albedo`, `Normal`). (Primarily for filename pattern)
* `[maptype]`: The standardized map type identifier (e.g., `COL` for Color/Albedo, `NRM` for Normal, `RGH` for Roughness). This is derived from the `standard_type` defined in the application's `FILE_TYPE_DEFINITIONS` (see `config/app_settings.json`) and may include a variant suffix if applicable. (Primarily for filename pattern)
* `[dimensions]`: Pixel dimensions (e.g., `2048x2048`).
* `[bitdepth]`: Output bit depth (e.g., `8bit`, `16bit`).
* `[category]`: Asset category determined by preset rules.

4
Documentation/02_Developer_Guide/04_Configuration_System_and_Presets.md
View File

@@ -62,8 +62,8 @@ The GUI includes a dedicated editor for modifying the `config/app_settings.json`
Preset files are the primary way to adapt the tool to new asset sources. Developers should use `Presets/_template.json` as a starting point. Key fields include:
* `supplier_name`: The name of the asset source (e.g., `"Poliigon"`). Used for output directory naming.
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a standard internal map type (defined in `config.py`).
* `target_type`: The standard internal map type (e.g., `"COL"`, `"NRM"`).
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a specific file type. The `target_type` for this mapping **must** be a key from `FILE_TYPE_DEFINITIONS` located in `config/app_settings.json`.
* `target_type`: The specific file type key from `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_NORM_GL"`, `"MAP_RGH"`). This replaces previous alias-based systems. The common aliases like "COL" or "NRM" are now derived from the `standard_type` property within `FILE_TYPE_DEFINITIONS` but are not used directly for `target_type`.
* `keywords`: A list of filename patterns (regex or fnmatch-style wildcards) used to identify this map type. The order of keywords within this list, and the order of dictionaries in the `map_type_mapping` list, determines the priority for assigning variant suffixes (`-1`, `-2`, etc.) when multiple files match the same `target_type`.
* `bit_depth_variants`: A dictionary mapping standard map types (e.g., `"NRM"`) to a pattern identifying its high bit-depth variant (e.g., `"*_NRM16*.tif"`). Files matching these patterns are prioritized over their standard counterparts.
* `map_bit_depth_rules`: Defines how to handle the bit depth of source maps. Can specify a default behavior (`"respect"` or `"force_8bit"`) and overrides for specific map types.

6
Documentation/02_Developer_Guide/05_Processing_Pipeline.md
View File

@@ -20,11 +20,11 @@ The pipeline steps are:
3. **File Inventory (`_inventory_and_classify_files`)**:
* Scans the contents of the *already prepared* temporary workspace.
* This step primarily inventories the files present. The *classification* (determining `item_type`, etc.) is taken directly from the input `SourceRule`.
* This step primarily inventories the files present. The *classification* (determining `item_type`, etc.) is taken directly from the input `SourceRule`. The `item_type` for each file (within the `FileRule` objects of the `SourceRule`) is expected to be a key from `Configuration.FILE_TYPE_DEFINITIONS`.
* Stores the file paths and their associated rules from the `SourceRule` in `self.classified_files`.
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` and the static `Configuration`. Overrides (like `supplier_identifier`, `asset_type`, `asset_name_override`) are taken directly from the `SourceRule`.
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` and the static `Configuration`. Overrides (like `supplier_identifier`, `asset_type`, `asset_name_override`) are taken directly from the `SourceRule`. The `asset_type` (within the `AssetRule` object of the `SourceRule`) is expected to be a key from `Configuration.ASSET_TYPE_DEFINITIONS`.
5. **Skip Check**:
* If the `overwrite` flag is `False`, checks if the final output directory already exists and contains `metadata.json`.
@@ -43,7 +43,7 @@ The pipeline steps are:
7. **Map Merging (`_merge_maps_from_source`)**:
* Iterates through `MAP_MERGE_RULES` in `Configuration`.
* Identifies required source maps by checking the `item_type_override` within the `SourceRule` (specifically in the `FileRule` for each file). Files with a base `item_type` of `"FILE_IGNORE"` are explicitly excluded from consideration.
* Identifies required source maps by checking the `item_type_override` within the `SourceRule` (specifically in the `FileRule` for each file). Both `item_type` and `item_type_override` are expected to be keys from `Configuration.FILE_TYPE_DEFINITIONS`. Files with a base `item_type` of `"FILE_IGNORE"` are explicitly excluded from consideration.
* Loads source channels, handling missing inputs with defaults from `Configuration` or `SourceRule`.
* Merges channels (`cv2.merge`).
* Determines output format/bit depth and saves the merged map.

49
Documentation/02_Developer_Guide/10_Coding_Conventions.md
View File

@@ -20,7 +20,7 @@ This document outlines the coding conventions and general practices followed wit
* Use Qt's signals and slots mechanism for communication between objects, especially across threads.
* Run long-running or blocking tasks in separate `QThread`s to keep the main UI thread responsive.
* Perform UI updates only from the main UI thread.
* **Configuration:** Core settings are managed in `config.py` (Python module). Supplier-specific rules are managed in JSON files (`Presets/`). The `Configuration` class handles loading and merging these.
* **Configuration:** Core application settings are defined in `config/app_settings.json`. Supplier-specific rules are managed in JSON files within the `Presets/` directory. The `Configuration` class (`configuration.py`) is responsible for loading `app_settings.json` and merging it with the selected preset file.
* **File Paths:** Use `pathlib.Path` objects for handling file system paths. Avoid using string manipulation for path joining or parsing.
* **Docstrings:** Write clear and concise docstrings for modules, classes, methods, and functions, explaining their purpose, arguments, and return values.
* **Comments:** Use comments to explain complex logic or non-obvious parts of the code.
@@ -31,4 +31,51 @@ This document outlines the coding conventions and general practices followed wit
* Use `UPPER_CASE` for constants.
* Use a leading underscore (`_`) for internal or "protected" methods/attributes.
## Terminology and Data Standards
To ensure consistency and clarity across the codebase, particularly concerning asset and file classifications, the following standards must be adhered to. These primarily revolve around definitions stored in `config/app_settings.json`.
### `FILE_TYPE_DEFINITIONS`
`FILE_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all file type identifiers used within the application.
* **`FileRule.item_type` and `FileRule.item_type_override`**: When defining or interpreting `SourceRule` objects (and their constituent `FileRule` instances), the `item_type` and `item_type_override` attributes **must** always use a key directly from `FILE_TYPE_DEFINITIONS`.
* Example: `file_rule.item_type = "MAP_COL"` (for a color map) or `file_rule.item_type = "MODEL_FBX"` (for an FBX model).
* **`standard_type` Property**: Each entry in `FILE_TYPE_DEFINITIONS` includes a `standard_type` property. This provides a common, often abbreviated, alias for the file type.
* Example: `FILE_TYPE_DEFINITIONS["MAP_COL"]["standard_type"]` might be `"COL"`.
* Example: `FILE_TYPE_DEFINITIONS["MAP_NORM_GL"]["standard_type"]` might be `"NRM"`.
* **Removal of `STANDARD_MAP_TYPES`**: The global constant `STANDARD_MAP_TYPES` (previously in `config.py`) has been **removed**. Standard map type aliases (e.g., "COL", "NRM", "RGH") are now derived dynamically from the `standard_type` property of the relevant entry in `FILE_TYPE_DEFINITIONS`.
* **`map_type` Usage**:
* **Filename Tokens**: When used as a token in output filename patterns (e.g., `[maptype]`), `map_type` is typically derived from the `standard_type` of the file's effective `item_type`. It may also include a variant suffix if applicable (e.g., "COL", "COL_var01").
* **General Classification**: For precise classification within code logic or rules, developers should refer to the full `FILE_TYPE_DEFINITIONS` key (e.g., `"MAP_COL"`, `"MAP_METAL"`). The `standard_type` can be used for broader categorization or when a common alias is needed.
* **`Map_type_Mapping.target_type` in Presets**: Within preset files (e.g., `Presets/Poliigon.json`), the `map_type_mapping` rules found inside a `FileRule`'s `map_processing_options` now use keys from `FILE_TYPE_DEFINITIONS` for the `target_type` field.
* Example:
```json
// Inside a FileRule in a preset
"map_processing_options": {
"map_type_mapping": {
"source_type_pattern": ".*ambient occlusion.*",
"target_type": "MAP_AO", // Uses FILE_TYPE_DEFINITIONS key
"source_channels": "RGB"
}
}
```
This replaces old aliases like `"AO"` or `"OCC"`.
* **`is_grayscale` Property**: `FILE_TYPE_DEFINITIONS` entries can now include an `is_grayscale` boolean property. This flag indicates whether the file type is inherently grayscale (e.g., a roughness map). It can be used by the processing engine to inform decisions about channel handling, compression, or specific image operations.
* Example: `FILE_TYPE_DEFINITIONS["MAP_RGH"]["is_grayscale"]` might be `true`.
### `ASSET_TYPE_DEFINITIONS`
Similarly, `ASSET_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all asset type identifiers.
* **`AssetRule.asset_type`, `AssetRule.asset_type_override`, and `AssetRule.asset_category`**: When defining or interpreting `SourceRule` objects (and their constituent `AssetRule` instances), the `asset_type`, `asset_type_override`, and `asset_category` attributes **must** always use a key directly from `ASSET_TYPE_DEFINITIONS`.
* Example: `asset_rule.asset_type = "SURFACE_3D"` or `asset_rule.asset_category = "FABRIC"`.
Adherence to these definitions ensures that terminology remains consistent throughout the application, from configuration to core logic and output.
Adhering to these conventions will make the codebase more consistent, easier to understand, and more maintainable for all contributors.