Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Major Terminogy unification and refactor [Needs thorough testing]

Browse Source
This commit is contained in:
Rusfort
2025-05-06 18:26:26 +02:00
parent 0a3100d448
commit ff548e902e
20 changed files with 689 additions and 356 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
Documentation/02_Developer_Guide/04_Configuration_System_and_Presets.md
View File

@@ -62,8 +62,8 @@ The GUI includes a dedicated editor for modifying the `config/app_settings.json`
Preset files are the primary way to adapt the tool to new asset sources. Developers should use `Presets/_template.json` as a starting point. Key fields include:
* `supplier_name`: The name of the asset source (e.g., `"Poliigon"`). Used for output directory naming.
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a standard internal map type (defined in `config.py`).
* `target_type`: The standard internal map type (e.g., `"COL"`, `"NRM"`).
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a specific file type. The `target_type` for this mapping **must** be a key from `FILE_TYPE_DEFINITIONS` located in `config/app_settings.json`.
* `target_type`: The specific file type key from `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_NORM_GL"`, `"MAP_RGH"`). This replaces previous alias-based systems. The common aliases like "COL" or "NRM" are now derived from the `standard_type` property within `FILE_TYPE_DEFINITIONS` but are not used directly for `target_type`.
* `keywords`: A list of filename patterns (regex or fnmatch-style wildcards) used to identify this map type. The order of keywords within this list, and the order of dictionaries in the `map_type_mapping` list, determines the priority for assigning variant suffixes (`-1`, `-2`, etc.) when multiple files match the same `target_type`.
* `bit_depth_variants`: A dictionary mapping standard map types (e.g., `"NRM"`) to a pattern identifying its high bit-depth variant (e.g., `"*_NRM16*.tif"`). Files matching these patterns are prioritized over their standard counterparts.
* `map_bit_depth_rules`: Defines how to handle the bit depth of source maps. Can specify a default behavior (`"respect"` or `"force_8bit"`) and overrides for specific map types.

6
Documentation/02_Developer_Guide/05_Processing_Pipeline.md
View File

@@ -20,11 +20,11 @@ The pipeline steps are:
3. **File Inventory (`_inventory_and_classify_files`)**:
* Scans the contents of the *already prepared* temporary workspace.
* This step primarily inventories the files present. The *classification* (determining `item_type`, etc.) is taken directly from the input `SourceRule`.
* This step primarily inventories the files present. The *classification* (determining `item_type`, etc.) is taken directly from the input `SourceRule`. The `item_type` for each file (within the `FileRule` objects of the `SourceRule`) is expected to be a key from `Configuration.FILE_TYPE_DEFINITIONS`.
* Stores the file paths and their associated rules from the `SourceRule` in `self.classified_files`.
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` and the static `Configuration`. Overrides (like `supplier_identifier`, `asset_type`, `asset_name_override`) are taken directly from the `SourceRule`.
* Determines the base asset name, category, and archetype using the explicit values provided in the input `SourceRule` and the static `Configuration`. Overrides (like `supplier_identifier`, `asset_type`, `asset_name_override`) are taken directly from the `SourceRule`. The `asset_type` (within the `AssetRule` object of the `SourceRule`) is expected to be a key from `Configuration.ASSET_TYPE_DEFINITIONS`.
5. **Skip Check**:
* If the `overwrite` flag is `False`, checks if the final output directory already exists and contains `metadata.json`.
@@ -43,7 +43,7 @@ The pipeline steps are:
7. **Map Merging (`_merge_maps_from_source`)**:
* Iterates through `MAP_MERGE_RULES` in `Configuration`.
* Identifies required source maps by checking the `item_type_override` within the `SourceRule` (specifically in the `FileRule` for each file). Files with a base `item_type` of `"FILE_IGNORE"` are explicitly excluded from consideration.
* Identifies required source maps by checking the `item_type_override` within the `SourceRule` (specifically in the `FileRule` for each file). Both `item_type` and `item_type_override` are expected to be keys from `Configuration.FILE_TYPE_DEFINITIONS`. Files with a base `item_type` of `"FILE_IGNORE"` are explicitly excluded from consideration.
* Loads source channels, handling missing inputs with defaults from `Configuration` or `SourceRule`.
* Merges channels (`cv2.merge`).
* Determines output format/bit depth and saves the merged map.

49
Documentation/02_Developer_Guide/10_Coding_Conventions.md
View File

@@ -20,7 +20,7 @@ This document outlines the coding conventions and general practices followed wit
* Use Qt's signals and slots mechanism for communication between objects, especially across threads.
* Run long-running or blocking tasks in separate `QThread`s to keep the main UI thread responsive.
* Perform UI updates only from the main UI thread.
* **Configuration:** Core settings are managed in `config.py` (Python module). Supplier-specific rules are managed in JSON files (`Presets/`). The `Configuration` class handles loading and merging these.
* **Configuration:** Core application settings are defined in `config/app_settings.json`. Supplier-specific rules are managed in JSON files within the `Presets/` directory. The `Configuration` class (`configuration.py`) is responsible for loading `app_settings.json` and merging it with the selected preset file.
* **File Paths:** Use `pathlib.Path` objects for handling file system paths. Avoid using string manipulation for path joining or parsing.
* **Docstrings:** Write clear and concise docstrings for modules, classes, methods, and functions, explaining their purpose, arguments, and return values.
* **Comments:** Use comments to explain complex logic or non-obvious parts of the code.
@@ -31,4 +31,51 @@ This document outlines the coding conventions and general practices followed wit
* Use `UPPER_CASE` for constants.
* Use a leading underscore (`_`) for internal or "protected" methods/attributes.
## Terminology and Data Standards
To ensure consistency and clarity across the codebase, particularly concerning asset and file classifications, the following standards must be adhered to. These primarily revolve around definitions stored in `config/app_settings.json`.
### `FILE_TYPE_DEFINITIONS`
`FILE_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all file type identifiers used within the application.
* **`FileRule.item_type` and `FileRule.item_type_override`**: When defining or interpreting `SourceRule` objects (and their constituent `FileRule` instances), the `item_type` and `item_type_override` attributes **must** always use a key directly from `FILE_TYPE_DEFINITIONS`.
* Example: `file_rule.item_type = "MAP_COL"` (for a color map) or `file_rule.item_type = "MODEL_FBX"` (for an FBX model).
* **`standard_type` Property**: Each entry in `FILE_TYPE_DEFINITIONS` includes a `standard_type` property. This provides a common, often abbreviated, alias for the file type.
* Example: `FILE_TYPE_DEFINITIONS["MAP_COL"]["standard_type"]` might be `"COL"`.
* Example: `FILE_TYPE_DEFINITIONS["MAP_NORM_GL"]["standard_type"]` might be `"NRM"`.
* **Removal of `STANDARD_MAP_TYPES`**: The global constant `STANDARD_MAP_TYPES` (previously in `config.py`) has been **removed**. Standard map type aliases (e.g., "COL", "NRM", "RGH") are now derived dynamically from the `standard_type` property of the relevant entry in `FILE_TYPE_DEFINITIONS`.
* **`map_type` Usage**:
* **Filename Tokens**: When used as a token in output filename patterns (e.g., `[maptype]`), `map_type` is typically derived from the `standard_type` of the file's effective `item_type`. It may also include a variant suffix if applicable (e.g., "COL", "COL_var01").
* **General Classification**: For precise classification within code logic or rules, developers should refer to the full `FILE_TYPE_DEFINITIONS` key (e.g., `"MAP_COL"`, `"MAP_METAL"`). The `standard_type` can be used for broader categorization or when a common alias is needed.
* **`Map_type_Mapping.target_type` in Presets**: Within preset files (e.g., `Presets/Poliigon.json`), the `map_type_mapping` rules found inside a `FileRule`'s `map_processing_options` now use keys from `FILE_TYPE_DEFINITIONS` for the `target_type` field.
* Example:
```json
// Inside a FileRule in a preset
"map_processing_options": {
"map_type_mapping": {
"source_type_pattern": ".*ambient occlusion.*",
"target_type": "MAP_AO", // Uses FILE_TYPE_DEFINITIONS key
"source_channels": "RGB"
}
}
```
This replaces old aliases like `"AO"` or `"OCC"`.
* **`is_grayscale` Property**: `FILE_TYPE_DEFINITIONS` entries can now include an `is_grayscale` boolean property. This flag indicates whether the file type is inherently grayscale (e.g., a roughness map). It can be used by the processing engine to inform decisions about channel handling, compression, or specific image operations.
* Example: `FILE_TYPE_DEFINITIONS["MAP_RGH"]["is_grayscale"]` might be `true`.
### `ASSET_TYPE_DEFINITIONS`
Similarly, `ASSET_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all asset type identifiers.
* **`AssetRule.asset_type`, `AssetRule.asset_type_override`, and `AssetRule.asset_category`**: When defining or interpreting `SourceRule` objects (and their constituent `AssetRule` instances), the `asset_type`, `asset_type_override`, and `asset_category` attributes **must** always use a key directly from `ASSET_TYPE_DEFINITIONS`.
* Example: `asset_rule.asset_type = "SURFACE_3D"` or `asset_rule.asset_category = "FABRIC"`.
Adherence to these definitions ensures that terminology remains consistent throughout the application, from configuration to core logic and output.
Adhering to these conventions will make the codebase more consistent, easier to understand, and more maintainable for all contributors.