Prototype > PreAlpha #67
6
.gitignore
vendored
6
.gitignore
vendored
@ -27,3 +27,9 @@ build/
|
||||
|
||||
# Ignore Windows thumbnail cache
|
||||
Thumbs.db
|
||||
gui/__pycache__
|
||||
__pycache__
|
||||
|
||||
|
||||
Testfiles/TestOutputs
|
||||
Testfiles_
|
||||
|
||||
5
.vscode/settings.json
vendored
5
.vscode/settings.json
vendored
@ -8,9 +8,6 @@
|
||||
".vscode": true,
|
||||
".vs": true,
|
||||
".lh": true,
|
||||
"__pycache__": true,
|
||||
"Deprecated-POC": true,
|
||||
"BlenderDocumentation": true,
|
||||
"PythonCheatsheats": true
|
||||
"__pycache__": true
|
||||
}
|
||||
}
|
||||
112
AUTOTEST_GUI_PLAN.md
Normal file
112
AUTOTEST_GUI_PLAN.md
Normal file
@ -0,0 +1,112 @@
|
||||
# Plan for Autotest GUI Mode Implementation
|
||||
|
||||
**I. Objective:**
|
||||
Create an `autotest.py` script that can launch the Asset Processor GUI headlessly, load a predefined asset (`.zip`), select a predefined preset, verify the predicted rule structure against an expected JSON, trigger processing to a predefined output directory, check the output, and analyze logs for errors or specific messages. This serves as a sanity check for core GUI-driven workflows.
|
||||
|
||||
**II. `TestFiles` Directory:**
|
||||
A new directory named `TestFiles` will be created in the project root (`c:/Users/Theis/Assetprocessor/Asset-Frameworker/TestFiles/`). This directory will house:
|
||||
* Sample asset `.zip` files for testing (e.g., `TestFiles/SampleAsset1.zip`).
|
||||
* Expected rule structure JSON files (e.g., `TestFiles/SampleAsset1_PresetX_expected_rules.json`).
|
||||
* A subdirectory for test outputs (e.g., `TestFiles/TestOutputs/`).
|
||||
|
||||
**III. `autotest.py` Script:**
|
||||
|
||||
1. **Location:** `c:/Users/Theis/Assetprocessor/Asset-Frameworker/autotest.py` (or `scripts/autotest.py`).
|
||||
2. **Command-Line Arguments (with defaults pointing to `TestFiles/`):**
|
||||
* `--zipfile`: Path to the test asset. Default: `TestFiles/default_test_asset.zip`.
|
||||
* `--preset`: Name of the preset. Default: `DefaultTestPreset`.
|
||||
* `--expectedrules`: Path to expected rules JSON. Default: `TestFiles/default_test_asset_rules.json`.
|
||||
* `--outputdir`: Path for processing output. Default: `TestFiles/TestOutputs/DefaultTestOutput`.
|
||||
* `--search` (optional): Log search term. Default: `None`.
|
||||
* `--additional-lines` (optional): Context lines for log search. Default: `0`.
|
||||
3. **Core Structure:**
|
||||
* Imports necessary modules from the main application and PySide6.
|
||||
* Adds project root to `sys.path` for imports.
|
||||
* `AutoTester` class:
|
||||
* **`__init__(self, app_instance: App)`:**
|
||||
* Stores `app_instance` and `main_window`.
|
||||
* Initializes `QEventLoop`.
|
||||
* Connects `app_instance.all_tasks_finished` to `self._on_all_tasks_finished`.
|
||||
* Loads expected rules from the `--expectedrules` file.
|
||||
* **`run_test(self)`:** Orchestrates the test steps sequentially:
|
||||
1. Load ZIP (`main_window.add_input_paths()`).
|
||||
2. Select Preset (`main_window.preset_editor_widget.editor_preset_list.setCurrentItem()`).
|
||||
3. Await Prediction (using `QTimer` to poll `main_window._pending_predictions`, manage with `QEventLoop`).
|
||||
4. Retrieve & Compare Rulelist:
|
||||
* Get actual rules: `main_window.unified_model.get_all_source_rules()`.
|
||||
* Convert actual rules to comparable dict (`_convert_rules_to_comparable()`).
|
||||
* Compare with loaded expected rules (`_compare_rules()`). If mismatch, log and fail.
|
||||
5. Start Processing (emit `main_window.start_backend_processing` with rules and output settings).
|
||||
6. Await Processing (use `QEventLoop` waiting for `_on_all_tasks_finished`).
|
||||
7. Check Output Path (verify existence of output dir, list contents, basic sanity checks like non-emptiness or presence of key asset folders).
|
||||
8. Retrieve & Analyze Logs (`main_window.log_console.log_console_output.toPlainText()`, filter by `--search`, check for tracebacks).
|
||||
9. Report result and call `cleanup_and_exit()`.
|
||||
* **`_check_prediction_status(self)`:** Slot for prediction polling timer.
|
||||
* **`_on_all_tasks_finished(self, processed_count, skipped_count, failed_count)`:** Slot for `App.all_tasks_finished` signal.
|
||||
* **`_convert_rules_to_comparable(self, source_rules_list: List[SourceRule]) -> dict`:** Converts `SourceRule` objects to the JSON structure defined below.
|
||||
* **`_compare_rules(self, actual_rules_data: dict, expected_rules_data: dict) -> bool`:** Implements Option 1 comparison logic:
|
||||
* Errors if an expected field is missing or its value mismatches.
|
||||
* Logs (but doesn't error on) fields present in actual but not in expected.
|
||||
* **`_process_and_display_logs(self, logs_text: str)`:** Handles log filtering/display.
|
||||
* **`cleanup_and_exit(self, success=True)`:** Quits `QCoreApplication` and `sys.exit()`.
|
||||
* `main()` function:
|
||||
* Parses CLI arguments.
|
||||
* Initializes `QApplication`.
|
||||
* Instantiates `main.App()` (does *not* show the GUI).
|
||||
* Instantiates `AutoTester(app_instance)`.
|
||||
* Uses `QTimer.singleShot(0, tester.run_test)` to start the test.
|
||||
* Runs `q_app.exec()`.
|
||||
|
||||
**IV. `expected_rules.json` Structure (Revised):**
|
||||
Located in `TestFiles/`. Example: `TestFiles/SampleAsset1_PresetX_expected_rules.json`.
|
||||
```json
|
||||
{
|
||||
"source_rules": [
|
||||
{
|
||||
"input_path": "SampleAsset1.zip",
|
||||
"supplier_identifier": "ExpectedSupplier",
|
||||
"preset_name": "PresetX",
|
||||
"assets": [
|
||||
{
|
||||
"asset_name": "AssetNameFromPrediction",
|
||||
"asset_type": "Prop",
|
||||
"files": [
|
||||
{
|
||||
"file_path": "relative/path/to/file1.png",
|
||||
"item_type": "MAP_COL",
|
||||
"target_asset_name_override": null
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**V. Mermaid Diagram of Autotest Flow:**
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start autotest.py with CLI Args (defaults to TestFiles/)] --> B{Setup Args & Logging};
|
||||
B --> C[Init QApplication & main.App (GUI Headless)];
|
||||
C --> D[Instantiate AutoTester(app_instance)];
|
||||
D --> E[QTimer.singleShot -> AutoTester.run_test()];
|
||||
|
||||
subgraph AutoTester.run_test()
|
||||
E --> F[Load Expected Rules from --expectedrules JSON];
|
||||
F --> G[Load ZIP (--zipfile) via main_window.add_input_paths()];
|
||||
G --> H[Select Preset (--preset) via main_window.preset_editor_widget];
|
||||
H --> I[Await Prediction (Poll main_window._pending_predictions via QTimer & QEventLoop)];
|
||||
I -- Prediction Done --> J[Get Actual Rules from main_window.unified_model];
|
||||
J --> K[Convert Actual Rules to Comparable JSON Structure];
|
||||
K --> L{Compare Actual vs Expected Rules (Option 1 Logic)};
|
||||
L -- Match --> M[Start Processing (Emit main_window.start_backend_processing with --outputdir)];
|
||||
L -- Mismatch --> ZFAIL[Log Mismatch & Call cleanup_and_exit(False)];
|
||||
M --> N[Await Processing (QEventLoop for App.all_tasks_finished signal)];
|
||||
N -- Processing Done --> O[Check Output Dir (--outputdir): Exists? Not Empty? Key Asset Folders?];
|
||||
O --> P[Retrieve & Analyze Logs (Search, Tracebacks)];
|
||||
P --> Q[Log Test Success & Call cleanup_and_exit(True)];
|
||||
end
|
||||
|
||||
ZFAIL --> ZEND[AutoTester.cleanup_and_exit() -> QCoreApplication.quit() & sys.exit()];
|
||||
Q --> ZEND;
|
||||
@ -1,291 +0,0 @@
|
||||
import bpy
|
||||
from pathlib import Path
|
||||
import time
|
||||
import os
|
||||
import math
|
||||
|
||||
# Try importing NumPy
|
||||
try:
|
||||
import numpy as np
|
||||
numpy_available = True
|
||||
# print("NumPy module found.") # Less verbose
|
||||
except ImportError:
|
||||
print("Warning: NumPy module not found. Median calc disabled, mean uses loop.")
|
||||
numpy_available = False
|
||||
|
||||
# --- Configuration ---
|
||||
ASSET_LIBRARY_NAME = "Nodes-Linked" # <<< Name of Asset Library in Prefs
|
||||
TEMPLATE_MATERIAL_NAME = "Template_PBRMaterial" # <<< Name of template Material in current file
|
||||
PLACEHOLDER_NODE_LABEL = "PBRSET_PLACEHOLDER" # <<< Label of placeholder node in template mat
|
||||
ASSET_NAME_PREFIX = "PBRSET_" # <<< Prefix of Node Group assets to process
|
||||
MATERIAL_NAME_PREFIX = "Mat_" # <<< Prefix for created Materials
|
||||
THUMBNAIL_PROPERTY_NAME = "thumbnail_filepath" # <<< Custom property name on Node Groups
|
||||
VALID_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff"}
|
||||
DERIVED_MAP_EXTENSIONS = ['.png', '.jpg', '.jpeg', '.tif', '.tiff']
|
||||
VIEWPORT_GAMMA = 0.4
|
||||
SCALED_SIZE = (32, 32) # Downscale target size for calculations
|
||||
|
||||
# --- >>> SET MATERIAL CREATION LIMIT HERE <<< ---
|
||||
# Max number of *new* materials created per run (0 = no limit)
|
||||
MATERIAL_CREATION_LIMIT = 900
|
||||
# ------------------------------------------------
|
||||
|
||||
# --- Helper Functions ---
|
||||
def find_node_by_label(node_tree, label, node_type=None):
|
||||
# Finds first node by label and optional type (using node.type)
|
||||
if not node_tree: return None
|
||||
for node in node_tree.nodes:
|
||||
if node.label and node.label == label:
|
||||
if node_type is None or node.type == node_type: return node
|
||||
return None
|
||||
|
||||
def calculate_value_from_image(image, target_size=(64, 64), mode='color', method='median'):
|
||||
# Calculates median/mean from downscaled image copy, cleans up temp image
|
||||
temp_img = None; #... (Full implementation from previous step) ...
|
||||
if not image: return None
|
||||
try:
|
||||
if not image.has_data:
|
||||
try: _ = len(image.pixels); image.update()
|
||||
except Exception: pass
|
||||
if not image.has_data: return None # Cannot proceed
|
||||
temp_img = image.copy()
|
||||
if not temp_img: return None
|
||||
temp_img.scale(target_size[0], target_size[1])
|
||||
try: _ = len(temp_img.pixels); temp_img.update()
|
||||
except Exception: pass # Ignore access error, check has_data
|
||||
if not temp_img.has_data: return None
|
||||
width=temp_img.size[0]; height=temp_img.size[1]; channels=temp_img.channels
|
||||
if width == 0 or height == 0 or channels == 0: return None
|
||||
pixels = temp_img.pixels[:]; result_value = None;
|
||||
if numpy_available: # Use NumPy
|
||||
np_pixels = np.array(pixels); num_elements = len(np_pixels); num_pixels_actual = num_elements // channels;
|
||||
if num_pixels_actual == 0: return None
|
||||
np_pixels = np_pixels[:num_pixels_actual * channels]; pixels_reshaped = np_pixels.reshape((num_pixels_actual, channels))
|
||||
if mode == 'color': # Color Median/Mean (NumPy)
|
||||
if channels < 3: return None
|
||||
calc_linear = np.median(pixels_reshaped[:, :3], axis=0) if method == 'median' else np.mean(pixels_reshaped[:, :3], axis=0)
|
||||
inv_gamma = 1.0 / VIEWPORT_GAMMA; calc_linear_clamped = np.clip(calc_linear, 0.0, None)
|
||||
calc_srgb_np = np.power(calc_linear_clamped, inv_gamma); calc_srgb_clamped = np.clip(calc_srgb_np, 0.0, 1.0)
|
||||
result_value = (calc_srgb_clamped[0], calc_srgb_clamped[1], calc_srgb_clamped[2], 1.0)
|
||||
elif mode == 'grayscale': # Grayscale Median/Mean (NumPy)
|
||||
calc_val = np.median(pixels_reshaped[:, 0]) if method == 'median' else np.mean(pixels_reshaped[:, 0])
|
||||
result_value = min(max(0.0, calc_val), 1.0)
|
||||
elif method == 'mean': # Fallback Mean Loop
|
||||
# print(" Calculating mean using standard loop...") # Verbose
|
||||
actual_len = len(pixels); #... (Mean loop logic) ...
|
||||
if actual_len == 0: return None; num_pixels_in_buffer=actual_len//channels; max_elements=num_pixels_in_buffer*channels
|
||||
if num_pixels_in_buffer == 0: return None
|
||||
if mode == 'color':
|
||||
sum_r,sum_g,sum_b = 0.0,0.0,0.0; step=channels
|
||||
for i in range(0, max_elements, step):
|
||||
if i+2 >= actual_len: break; sum_r+=pixels[i]; sum_g+=pixels[i+1]; sum_b+=pixels[i+2]
|
||||
avg_r_lin,avg_g_lin,avg_b_lin = sum_r/num_pixels_in_buffer, sum_g/num_pixels_in_buffer, sum_b/num_pixels_in_buffer
|
||||
inv_gamma = 1.0/VIEWPORT_GAMMA
|
||||
avg_r_srgb,avg_g_srgb,avg_b_srgb = min(max(0.0,pow(max(0.0,avg_r_lin),inv_gamma)),1.0), min(max(0.0,pow(max(0.0,avg_g_lin),inv_gamma)),1.0), min(max(0.0,pow(max(0.0,avg_b_lin),inv_gamma)),1.0)
|
||||
result_value = (avg_r_srgb, avg_g_srgb, avg_b_srgb, 1.0)
|
||||
elif mode == 'grayscale':
|
||||
sum_val=0.0; step=channels
|
||||
for i in range(0, max_elements, step): sum_val+=pixels[i]
|
||||
result_value = min(max(0.0, sum_val/num_pixels_in_buffer), 1.0)
|
||||
else: print(" Error: NumPy required for median calculation."); return None
|
||||
return result_value
|
||||
except Exception as e: print(f" Error during value calculation for '{image.name}': {e}"); return None
|
||||
finally: # Cleanup
|
||||
if temp_img:
|
||||
try: bpy.data.images.remove(temp_img, do_unlink=True)
|
||||
except Exception: pass # Ignore cleanup errors
|
||||
|
||||
|
||||
# --- Main Function ---
|
||||
def create_materials_for_library_assets(library_name):
|
||||
start_time = time.time(); print(f"--- Starting Material Creation for Library '{library_name}' ---")
|
||||
print(f"Material Creation Limit per run: {'Unlimited' if MATERIAL_CREATION_LIMIT <= 0 else MATERIAL_CREATION_LIMIT}")
|
||||
# (Prerequisite checks...)
|
||||
template_mat=bpy.data.materials.get(TEMPLATE_MATERIAL_NAME); #... etc ...
|
||||
if not template_mat or not template_mat.use_nodes or not find_node_by_label(template_mat.node_tree, PLACEHOLDER_NODE_LABEL, 'GROUP'): print("Template Prereq Failed."); return
|
||||
library=bpy.context.preferences.filepaths.asset_libraries.get(library_name); #... etc ...
|
||||
if not library or not Path(bpy.path.abspath(library.path)).exists(): print("Library Prereq Failed."); return
|
||||
print(f"Found template material and library path...")
|
||||
|
||||
# (File scanning...)
|
||||
materials_created=0; materials_skipped=0; nodegroups_processed=0; link_errors=0; files_to_process=[]; library_path_obj=Path(bpy.path.abspath(library.path))
|
||||
#... (populate files_to_process) ...
|
||||
if library_path_obj.is_dir():
|
||||
for item in library_path_obj.iterdir():
|
||||
if item.is_file() and item.suffix.lower() == '.blend': files_to_process.append(str(item))
|
||||
if not files_to_process: print(f"Warning: No .blend files found in dir: {library_path_obj}")
|
||||
elif library_path_obj.is_file() and library_path_obj.suffix.lower() == '.blend':
|
||||
files_to_process.append(str(library_path_obj))
|
||||
else: print(f"Error: Library path not dir or .blend: {library_path_obj}"); return
|
||||
print(f"Found {len(files_to_process)} .blend file(s) to inspect.")
|
||||
|
||||
# Initialize counters and flag for limit
|
||||
created_in_this_run = 0
|
||||
limit_reached_flag = False
|
||||
|
||||
for blend_file_path in files_to_process: # ... (inspect loop) ...
|
||||
print(f"\nInspecting library file: {os.path.basename(blend_file_path)}...")
|
||||
potential_nodegroups = []; # ... (inspection logic) ...
|
||||
try:
|
||||
with bpy.data.libraries.load(blend_file_path, link=False) as (data_from, data_to): potential_nodegroups = list(data_from.node_groups)
|
||||
except Exception as e_load_inspect: print(f" Error inspecting file '{blend_file_path}': {e_load_inspect}"); continue
|
||||
print(f" Found {len(potential_nodegroups)} NGs. Checking for '{ASSET_NAME_PREFIX}'...")
|
||||
|
||||
|
||||
for asset_nodegroup_name in potential_nodegroups: # ... (NG loop) ...
|
||||
if not asset_nodegroup_name.startswith(ASSET_NAME_PREFIX): continue
|
||||
nodegroups_processed += 1
|
||||
base_name = asset_nodegroup_name.removeprefix(ASSET_NAME_PREFIX)
|
||||
material_name = f"{MATERIAL_NAME_PREFIX}{base_name}"
|
||||
if bpy.data.materials.get(material_name): materials_skipped += 1; continue
|
||||
|
||||
linked_nodegroup = None; preview_path = None
|
||||
try: # --- Start Main Processing Block for NG ---
|
||||
# (Linking logic...)
|
||||
existing_group = bpy.data.node_groups.get(asset_nodegroup_name); #... etc linking ...
|
||||
is_correctly_linked = (existing_group and existing_group.library and bpy.path.abspath(existing_group.library.filepath) == blend_file_path)
|
||||
if is_correctly_linked: linked_nodegroup = existing_group
|
||||
else: # Link it
|
||||
with bpy.data.libraries.load(blend_file_path, link=True, relative=False) as (data_from, data_to):
|
||||
if asset_nodegroup_name in data_from.node_groups: data_to.node_groups = [asset_nodegroup_name]
|
||||
else: print(f" Error: NG '{asset_nodegroup_name}' not found during link."); continue # Skip NG
|
||||
linked_nodegroup = bpy.data.node_groups.get(asset_nodegroup_name)
|
||||
if not linked_nodegroup or not linked_nodegroup.library: print(f" Error: NG '{asset_nodegroup_name}' link failed."); linked_nodegroup = None; link_errors += 1
|
||||
if not linked_nodegroup: print(f" Failed link NG '{asset_nodegroup_name}'. Skip."); continue # Skip NG
|
||||
|
||||
preview_path = linked_nodegroup.get(THUMBNAIL_PROPERTY_NAME) # Path to COL-1 1K
|
||||
|
||||
# (Duplicate, Rename, Replace Placeholder...)
|
||||
new_material = template_mat.copy(); #... checks ...
|
||||
if not new_material: print(f" Error: Failed copy template mat. Skip."); continue
|
||||
new_material.name = material_name
|
||||
if not new_material.use_nodes or not new_material.node_tree: print(f" Error: New mat '{material_name}' no nodes."); continue
|
||||
placeholder_node = find_node_by_label(new_material.node_tree, PLACEHOLDER_NODE_LABEL, 'GROUP'); #... checks ...
|
||||
if not placeholder_node: print(f" Error: Placeholder '{PLACEHOLDER_NODE_LABEL}' not found."); continue
|
||||
placeholder_node.node_tree = linked_nodegroup
|
||||
print(f" Created material '{material_name}' and linked NG '{linked_nodegroup.name}'.")
|
||||
|
||||
# --- Load base COL-1 image once ---
|
||||
thumbnail_image = None
|
||||
if preview_path and Path(preview_path).is_file():
|
||||
try: thumbnail_image = bpy.data.images.load(preview_path, check_existing=True)
|
||||
except Exception as e_load_base: print(f" Error loading base thumbnail '{preview_path}': {e_load_base}")
|
||||
|
||||
# --- Set Viewport Color (Median) ---
|
||||
median_color = None
|
||||
if thumbnail_image: median_color = calculate_value_from_image(thumbnail_image, target_size=SCALED_SIZE, mode='color', method='median')
|
||||
if median_color: new_material.diffuse_color = median_color; print(f" Set viewport color: {median_color[:3]}")
|
||||
else: print(f" Warn: Could not set viewport color.")
|
||||
|
||||
|
||||
# --- Determine Paths and Metal Map Existence ---
|
||||
roughness_path = None; metallic_path = None; metal_map_found = False; #... etc ...
|
||||
if preview_path and "_COL-1" in preview_path:
|
||||
try: # ... path derivation logic ...
|
||||
base_path_obj=Path(preview_path); directory=base_path_obj.parent; base_stem=base_path_obj.stem
|
||||
if "_COL-1" in base_stem:
|
||||
rough_stem=base_stem.replace("_COL-1", "_ROUGH")
|
||||
for ext in DERIVED_MAP_EXTENSIONS:
|
||||
potential_path=directory/f"{rough_stem}{ext}";
|
||||
if potential_path.is_file(): roughness_path=str(potential_path); break
|
||||
metal_stem=base_stem.replace("_COL-1", "_METAL")
|
||||
for ext in DERIVED_MAP_EXTENSIONS:
|
||||
potential_path=directory/f"{metal_stem}{ext}";
|
||||
if potential_path.is_file(): metallic_path=str(potential_path); metal_map_found=True; break
|
||||
except Exception as e_derive: print(f" Error deriving paths: {e_derive}")
|
||||
if not metal_map_found: print(f" Info: No METAL map found. Assuming Spec/Gloss.")
|
||||
|
||||
# --- Set Viewport Roughness (Median, Conditional Inversion) ---
|
||||
median_roughness = None; # ... etc ...
|
||||
if roughness_path:
|
||||
try: rough_img = bpy.data.images.load(roughness_path, check_existing=True)
|
||||
except Exception as e_load_rough: print(f" Error loading rough image: {e_load_rough}")
|
||||
if rough_img: median_roughness = calculate_value_from_image(rough_img, target_size=SCALED_SIZE, mode='grayscale', method='median')
|
||||
else: print(f" Error: load None for rough path.")
|
||||
if median_roughness is not None:
|
||||
final_roughness_value = median_roughness
|
||||
if not metal_map_found: final_roughness_value = 1.0 - median_roughness; print(f" Inverting ROUGH->Gloss: {median_roughness:.3f} -> {final_roughness_value:.3f}")
|
||||
new_material.roughness = min(max(0.0, final_roughness_value), 1.0); print(f" Set viewport roughness: {new_material.roughness:.3f}")
|
||||
else: print(f" Warn: Could not set viewport roughness.")
|
||||
|
||||
|
||||
# --- Set Viewport Metallic (Median) ---
|
||||
median_metallic = None; # ... etc ...
|
||||
if metal_map_found:
|
||||
try: metal_img = bpy.data.images.load(metallic_path, check_existing=True)
|
||||
except Exception as e_load_metal: print(f" Error loading metal image: {e_load_metal}")
|
||||
if metal_img: median_metallic = calculate_value_from_image(metal_img, target_size=SCALED_SIZE, mode='grayscale', method='median')
|
||||
else: print(f" Error: load None for metal path.")
|
||||
if median_metallic is not None: new_material.metallic = median_metallic; print(f" Set viewport metallic: {median_metallic:.3f}")
|
||||
else: new_material.metallic = 0.0; # Default
|
||||
if metal_map_found: print(f" Warn: Could not calc viewport metallic. Set 0.0.")
|
||||
else: print(f" Set viewport metallic to default: 0.0")
|
||||
|
||||
|
||||
# --- Mark Material as Asset ---
|
||||
mat_asset_data = None; # ... (logic remains same) ...
|
||||
try: # ... asset marking ...
|
||||
if not new_material.asset_data: new_material.asset_mark(); print(f" Marked material as asset.")
|
||||
mat_asset_data = new_material.asset_data
|
||||
except Exception as e_asset: print(f" Error marking mat asset: {e_asset}")
|
||||
|
||||
# --- Copy Asset Tags ---
|
||||
if mat_asset_data and linked_nodegroup.asset_data: # ... (logic remains same) ...
|
||||
try: # ... tag copying ...
|
||||
source_tags=linked_nodegroup.asset_data.tags; target_tags=mat_asset_data.tags
|
||||
tags_copied_count=0; existing_target_tag_names={t.name for t in target_tags}
|
||||
for src_tag in source_tags:
|
||||
if src_tag.name not in existing_target_tag_names: target_tags.new(name=src_tag.name); tags_copied_count += 1
|
||||
if tags_copied_count > 0: print(f" Copied {tags_copied_count} asset tags.")
|
||||
except Exception as e_tags: print(f" Error copying tags: {e_tags}")
|
||||
|
||||
# --- Set Custom Preview for Material ---
|
||||
if preview_path and Path(preview_path).is_file(): # ... (logic remains same) ...
|
||||
try: # ... preview setting ...
|
||||
with bpy.context.temp_override(id=new_material): bpy.ops.ed.lib_id_load_custom_preview(filepath=preview_path)
|
||||
except RuntimeError as e_op: print(f" Error running preview op for mat '{new_material.name}': {e_op}")
|
||||
except Exception as e_prev: print(f" Unexpected preview error for mat: {e_prev}")
|
||||
elif preview_path: print(f" Warn: Thumb path not found for preview step: '{preview_path}'")
|
||||
|
||||
|
||||
# --- Increment Counters & Check Limit ---
|
||||
materials_created += 1 # Overall counter for summary
|
||||
created_in_this_run += 1 # Counter for this run's limit
|
||||
|
||||
# Check limit AFTER successful creation
|
||||
if MATERIAL_CREATION_LIMIT > 0 and created_in_this_run >= MATERIAL_CREATION_LIMIT:
|
||||
print(f"\n--- Material Creation Limit ({MATERIAL_CREATION_LIMIT}) Reached ---")
|
||||
limit_reached_flag = True
|
||||
break # Exit inner loop
|
||||
|
||||
except Exception as e: # Catch errors for the whole NG processing block
|
||||
print(f" An unexpected error occurred processing NG '{asset_nodegroup_name}': {e}")
|
||||
# --- End Main Processing Block for NG ---
|
||||
|
||||
# Check flag to stop outer loop
|
||||
if limit_reached_flag:
|
||||
print("Stopping library file iteration due to limit.")
|
||||
break # Exit outer loop
|
||||
|
||||
# (Completion summary...)
|
||||
end_time = time.time(); duration = end_time - start_time; print("\n--- Material Creation Finished ---"); # ... etc ...
|
||||
print(f"Duration: {duration:.2f} seconds")
|
||||
print(f"Summary: Processed {nodegroups_processed} NGs. Created {materials_created} Mats this run. Skipped {materials_skipped}. Link Errors {link_errors}.")
|
||||
if limit_reached_flag: print(f"NOTE: Script stopped early due to creation limit ({MATERIAL_CREATION_LIMIT}). Run again to process more.")
|
||||
|
||||
# --- How to Run ---
|
||||
# 1. Rerun Script 1 to add "thumbnail_filepath" property.
|
||||
# 2. Setup Asset Library in Prefs. Set ASSET_LIBRARY_NAME below.
|
||||
# 3. In current file, create "Template_PBRMaterial" with "PBRSET_PLACEHOLDER" node.
|
||||
# 4. Set MATERIAL_CREATION_LIMIT in Config section above (0 for unlimited).
|
||||
# 5. Paste script & Run (Alt+P).
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Only need ASSET_LIBRARY_NAME configuration here now
|
||||
if ASSET_LIBRARY_NAME == "My Asset Library": # Default check
|
||||
print("\nERROR: Please update the 'ASSET_LIBRARY_NAME' variable in the script's Configuration section.")
|
||||
print(" Set it to the name of your asset library in Blender Preferences before running.\n")
|
||||
elif not bpy.data.materials.get(TEMPLATE_MATERIAL_NAME):
|
||||
print(f"\nERROR: Template material '{TEMPLATE_MATERIAL_NAME}' not found in current file.\n")
|
||||
else:
|
||||
create_materials_for_library_assets(ASSET_LIBRARY_NAME)
|
||||
@ -1,988 +0,0 @@
|
||||
# Full script - PBR Texture Importer V4 (Manifest, Auto-Save/Reload, Aspect Ratio, Asset Tags)
|
||||
|
||||
import bpy
|
||||
import os # For auto-save rename/remove
|
||||
from pathlib import Path
|
||||
import time
|
||||
import base64
|
||||
import numpy as np # For stats calculation
|
||||
import json # For manifest handling
|
||||
import re # For parsing scaling string
|
||||
|
||||
# --- USER CONFIGURATION ---
|
||||
# File Paths & Templates
|
||||
texture_root_directory = r"G:\02 Content\10-19 Content\13 Textures Power of Two\13.00" # <<< CHANGE THIS PATH!
|
||||
PARENT_TEMPLATE_NAME = "Template_PBRSET" # Name of the parent node group template
|
||||
CHILD_TEMPLATE_NAME = "Template_PBRTYPE" # Name of the child node group template
|
||||
|
||||
# Processing Limits & Intervals
|
||||
MAX_NEW_GROUPS_PER_RUN = 1000 # Max NEW parent groups created per run before stopping
|
||||
SAVE_INTERVAL = 25 # Auto-save interval during NEW group creation (every N groups)
|
||||
|
||||
# Features & Behavior
|
||||
AUTO_SAVE_ENABLED = True # Enable periodic auto-saving (main file + manifest) during processing?
|
||||
AUTO_RELOAD_ON_FINISH = True # Save and reload the blend file upon successful script completion?
|
||||
|
||||
# Naming & Structure Conventions
|
||||
VALID_EXTENSIONS = {".jpg", ".jpeg", ".png", ".tif", ".tiff"} # Allowed texture file types
|
||||
RESOLUTION_LABELS = ["1k", "2k", "4k", "8k"] # Expected resolution labels (LOWEST FIRST for aspect/tag calc)
|
||||
SG_VALUE_NODE_LABEL = "SpecularGlossy" # Label for the Specular/Glossy value node in parent template
|
||||
HISTOGRAM_NODE_PREFIX = "Histogram-" # Prefix for Combine XYZ nodes storing stats (e.g., "Histogram-ROUGH")
|
||||
ASPECT_RATIO_NODE_LABEL = "AspectRatioCorrection" # Label for the Value node storing the aspect ratio correction factor
|
||||
|
||||
# Texture Map Properties
|
||||
PBR_COLOR_SPACE_MAP = { # Map PBR type (from filename) to Blender color space
|
||||
"AO": "sRGB", "COL-1": "sRGB", "COL-2": "sRGB", "COL-3": "sRGB",
|
||||
"DISP": "Non-Color", "NRM": "Non-Color", "REFL": "Non-Color", "ROUGH": "Non-Color",
|
||||
"METAL": "Non-Color", "FUZZ": "Non-Color", "MASK": "Non-Color", "SSS": "sRGB",
|
||||
}
|
||||
DEFAULT_COLOR_SPACE = "sRGB" # Fallback color space if PBR type not in map
|
||||
|
||||
# --- END USER CONFIGURATION ---
|
||||
|
||||
|
||||
# --- Helper Functions ---
|
||||
|
||||
def parse_texture_filename(filename_stem):
|
||||
"""Parses texture filename stem based on expected convention."""
|
||||
parts = filename_stem.split('_');
|
||||
# Expecting Tag_Groupname_Resolution_Scaling_PBRType
|
||||
if len(parts) == 5:
|
||||
return {"Tag": parts[0], "Groupname": parts[1], "Resolution": parts[2], "Scaling": parts[3], "PBRType": parts[4]}
|
||||
else:
|
||||
print(f" Warn: Skip '{filename_stem}' - Expected 5 parts, found {len(parts)}.");
|
||||
return None
|
||||
|
||||
def find_nodes_by_label(node_tree, label, node_type=None):
|
||||
"""Finds ALL nodes in a node tree matching the label and optionally type."""
|
||||
if not node_tree:
|
||||
return []
|
||||
matching_nodes = []
|
||||
for node in node_tree.nodes:
|
||||
if node.label and node.label == label:
|
||||
if node_type is None or node.type == node_type:
|
||||
matching_nodes.append(node)
|
||||
return matching_nodes
|
||||
|
||||
def encode_name_b64(name_str):
|
||||
"""Encodes a string using URL-safe Base64 for node group names."""
|
||||
try:
|
||||
return base64.urlsafe_b64encode(name_str.encode('utf-8')).decode('ascii')
|
||||
except Exception as e:
|
||||
print(f" Error base64 encoding '{name_str}': {e}");
|
||||
return name_str # Fallback to original name on error
|
||||
|
||||
def calculate_image_stats(image):
|
||||
"""Calculates Min, Max, Median of the first channel of a Blender image."""
|
||||
if not image: return None
|
||||
pixels_arr, value_channel_arr, result = None, None, None
|
||||
try:
|
||||
width = image.size[0]; height = image.size[1]; channels = image.channels
|
||||
if width == 0 or height == 0 or channels == 0:
|
||||
print(f" Warn: Invalid dims for '{image.name}'. Skip stats."); return None
|
||||
actual_len = len(image.pixels); expected_len = width * height * channels
|
||||
if expected_len != actual_len:
|
||||
print(f" Warn: Pixel buffer mismatch for '{image.name}'. Skip stats."); return None
|
||||
if actual_len == 0: return None
|
||||
|
||||
pixels_arr = np.fromiter(image.pixels, dtype=np.float32, count=actual_len)
|
||||
|
||||
if channels == 1: value_channel_arr = pixels_arr
|
||||
elif channels >= 2: value_channel_arr = pixels_arr[0::channels]
|
||||
else: return None
|
||||
|
||||
if value_channel_arr is None or value_channel_arr.size == 0:
|
||||
print(f" Warn: No value channel for '{image.name}'. Skip stats."); return None
|
||||
|
||||
min_val = float(np.min(value_channel_arr))
|
||||
max_val = float(np.max(value_channel_arr))
|
||||
median_val = float(np.median(value_channel_arr))
|
||||
result = (min_val, max_val, median_val)
|
||||
|
||||
except MemoryError:
|
||||
print(f" Error: Not enough memory for stats calc on '{image.name}'.")
|
||||
except Exception as e:
|
||||
print(f" Error during stats calc for '{image.name}': {e}");
|
||||
import traceback; traceback.print_exc()
|
||||
finally:
|
||||
# Explicitly delete potentially large numpy arrays
|
||||
if 'value_channel_arr' in locals() and value_channel_arr is not None:
|
||||
try:
|
||||
del value_channel_arr
|
||||
except NameError:
|
||||
pass # Ignore if already gone
|
||||
if 'pixels_arr' in locals() and pixels_arr is not None:
|
||||
try:
|
||||
del pixels_arr
|
||||
except NameError:
|
||||
pass # Ignore if already gone
|
||||
return result
|
||||
|
||||
def calculate_aspect_ratio_factor(image_width, image_height, scaling_string):
|
||||
"""Calculates the X-axis UV scaling factor based on image dims and scaling string."""
|
||||
if image_height <= 0:
|
||||
print(" Warn: Image height is zero, cannot calculate aspect ratio. Returning 1.0.")
|
||||
return 1.0 # Return 1.0 if height is invalid
|
||||
|
||||
# Calculate the actual aspect ratio of the image file
|
||||
current_aspect_ratio = image_width / image_height
|
||||
|
||||
# Check the scaling string
|
||||
if scaling_string.upper() == "EVEN":
|
||||
# 'EVEN' means uniform scaling was applied (or none needed).
|
||||
# The correction factor is the image's own aspect ratio.
|
||||
return current_aspect_ratio
|
||||
else:
|
||||
# Handle non-uniform scaling cases ("Xnnn", "Ynnn")
|
||||
match = re.match(r"([XY])(\d+)", scaling_string, re.IGNORECASE)
|
||||
if not match:
|
||||
print(f" Warn: Invalid Scaling string format '{scaling_string}'. Returning current ratio {current_aspect_ratio:.4f} as fallback.")
|
||||
# Fallback to the image's own ratio if scaling string is invalid
|
||||
return current_aspect_ratio
|
||||
|
||||
axis = match.group(1).upper()
|
||||
try:
|
||||
amount = int(match.group(2))
|
||||
if amount <= 0:
|
||||
print(f" Warn: Zero or negative Amount in Scaling string '{scaling_string}'. Returning current ratio {current_aspect_ratio:.4f}.")
|
||||
return current_aspect_ratio
|
||||
except ValueError:
|
||||
print(f" Warn: Invalid Amount in Scaling string '{scaling_string}'. Returning current ratio {current_aspect_ratio:.4f}.")
|
||||
return current_aspect_ratio
|
||||
|
||||
# Apply the non-uniform correction formula
|
||||
factor = current_aspect_ratio # Default to current ratio in case of issues below
|
||||
scaling_factor_percent = amount / 100.0
|
||||
try:
|
||||
if axis == 'X':
|
||||
if scaling_factor_percent == 0: raise ZeroDivisionError
|
||||
factor = current_aspect_ratio / scaling_factor_percent
|
||||
elif axis == 'Y':
|
||||
factor = current_aspect_ratio * scaling_factor_percent
|
||||
# No 'else' needed due to regex structure
|
||||
|
||||
except ZeroDivisionError:
|
||||
print(f" Warn: Division by zero during factor calculation. Returning current ratio {current_aspect_ratio:.4f}.")
|
||||
return current_aspect_ratio
|
||||
|
||||
return factor
|
||||
|
||||
# --- Manifest Helper Functions ---
|
||||
def get_manifest_path(context_filepath):
|
||||
"""Gets the expected path for the manifest JSON file based on blend filepath."""
|
||||
if not context_filepath:
|
||||
return None
|
||||
blend_path = Path(context_filepath)
|
||||
manifest_filename = f"{blend_path.stem}_manifest.json"
|
||||
return blend_path.parent / manifest_filename
|
||||
|
||||
def load_manifest(manifest_path):
|
||||
"""Loads the manifest data from the JSON file."""
|
||||
if not manifest_path or not manifest_path.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(manifest_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
print(f" Loaded manifest from: {manifest_path.name}")
|
||||
return data
|
||||
except json.JSONDecodeError:
|
||||
print(f"!!! ERROR: Manifest file '{manifest_path.name}' is corrupted. Starting fresh. !!!")
|
||||
return {}
|
||||
except Exception as e:
|
||||
print(f"!!! ERROR: Could not load manifest file '{manifest_path.name}': {e} !!!")
|
||||
return {}
|
||||
|
||||
def save_manifest(manifest_path, data):
|
||||
"""Saves the manifest data to the JSON file."""
|
||||
if not manifest_path:
|
||||
return False
|
||||
try:
|
||||
with open(manifest_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"!!!!!!!!!!!!!!!!!!!\n!!! Manifest save FAILED: {e} !!!\n!!!!!!!!!!!!!!!!!!!")
|
||||
return False
|
||||
|
||||
# --- Auto-Save Helper Function ---
|
||||
def perform_safe_autosave(manifest_path, manifest_data):
|
||||
"""Performs a safe auto-save of the main blend file and manifest."""
|
||||
blend_filepath = bpy.data.filepath
|
||||
if not blend_filepath or not manifest_path:
|
||||
print(" Skipping auto-save: Blend file is not saved.")
|
||||
return False
|
||||
|
||||
print(f"\n--- Attempting Auto-Save ({time.strftime('%H:%M:%S')}) ---")
|
||||
blend_path = Path(blend_filepath)
|
||||
manifest_path_obj = Path(manifest_path) # Ensure it's a Path object
|
||||
|
||||
blend_bak_path = blend_path.with_suffix('.blend.bak')
|
||||
manifest_bak_path = manifest_path_obj.with_suffix('.json.bak')
|
||||
|
||||
# 1. Delete old backups if they exist
|
||||
try:
|
||||
if blend_bak_path.exists():
|
||||
blend_bak_path.unlink()
|
||||
if manifest_bak_path.exists():
|
||||
manifest_bak_path.unlink()
|
||||
except OSError as e:
|
||||
print(f" Warn: Could not delete old backup file: {e}")
|
||||
# Continue anyway, renaming might still work
|
||||
|
||||
# 2. Rename current files to backup
|
||||
renamed_blend = False
|
||||
renamed_manifest = False
|
||||
try:
|
||||
if blend_path.exists():
|
||||
os.rename(blend_path, blend_bak_path)
|
||||
renamed_blend = True
|
||||
# print(f" Renamed '{blend_path.name}' to '{blend_bak_path.name}'") # Optional verbose log
|
||||
if manifest_path_obj.exists():
|
||||
os.rename(manifest_path_obj, manifest_bak_path)
|
||||
renamed_manifest = True
|
||||
# print(f" Renamed '{manifest_path_obj.name}' to '{manifest_bak_path.name}'") # Optional verbose log
|
||||
except OSError as e:
|
||||
print(f"!!! ERROR: Failed to rename files for backup: {e} !!!")
|
||||
# Attempt to roll back renames if only one succeeded
|
||||
if renamed_blend and not renamed_manifest and blend_bak_path.exists():
|
||||
print(f" Attempting rollback: Renaming {blend_bak_path.name} back...")
|
||||
try:
|
||||
os.rename(blend_bak_path, blend_path)
|
||||
except OSError as rb_e:
|
||||
print(f" Rollback rename of blend file FAILED: {rb_e}")
|
||||
if renamed_manifest and not renamed_blend and manifest_bak_path.exists():
|
||||
print(f" Attempting rollback: Renaming {manifest_bak_path.name} back...")
|
||||
try:
|
||||
os.rename(manifest_bak_path, manifest_path_obj)
|
||||
except OSError as rb_e:
|
||||
print(f" Rollback rename of manifest file FAILED: {rb_e}")
|
||||
print("--- Auto-Save ABORTED ---")
|
||||
return False
|
||||
|
||||
# 3. Save new main blend file
|
||||
save_blend_success = False
|
||||
try:
|
||||
bpy.ops.wm.save_mainfile()
|
||||
print(f" Saved main blend file: {blend_path.name}")
|
||||
save_blend_success = True
|
||||
except Exception as e:
|
||||
print(f"!!!!!!!!!!!!!!!!!!!!!!!!!\n!!! Auto-Save FAILED (Blend File Save): {e} !!!\n!!!!!!!!!!!!!!!!!!!!!!!!!")
|
||||
# Attempt to restore from backup
|
||||
print(" Attempting to restore from backup...")
|
||||
try:
|
||||
if blend_bak_path.exists():
|
||||
os.rename(blend_bak_path, blend_path)
|
||||
if manifest_bak_path.exists():
|
||||
os.rename(manifest_bak_path, manifest_path_obj)
|
||||
print(" Restored from backup.")
|
||||
except OSError as re:
|
||||
print(f"!!! CRITICAL: Failed to restore from backup after save failure: {re} !!!")
|
||||
print(f"!!! Please check for '.bak' files manually in: {blend_path.parent} !!!")
|
||||
print("--- Auto-Save ABORTED ---")
|
||||
return False
|
||||
|
||||
# 4. Save new manifest file (only if blend save succeeded)
|
||||
if save_blend_success:
|
||||
if save_manifest(manifest_path, manifest_data):
|
||||
print(f" Saved manifest file: {manifest_path_obj.name}")
|
||||
print("--- Auto-Save Successful ---")
|
||||
return True
|
||||
else:
|
||||
# Manifest save failed, but blend file is okay. Warn user.
|
||||
print("!!! WARNING: Auto-save completed for blend file, but manifest save FAILED. Manifest may be out of sync. !!!")
|
||||
return True # Still counts as 'completed' in terms of blend file safety
|
||||
|
||||
return False # Should not be reached
|
||||
|
||||
# --- Asset Tagging Helper Functions ---
|
||||
|
||||
def add_tag_if_new(asset_data, tag_name):
|
||||
"""Adds a tag to the asset data if it's not None/empty and doesn't already exist."""
|
||||
if not asset_data or not tag_name or not isinstance(tag_name, str) or tag_name.strip() == "":
|
||||
return False # Invalid input
|
||||
cleaned_tag_name = tag_name.strip() # Remove leading/trailing whitespace
|
||||
if not cleaned_tag_name:
|
||||
return False # Don't add empty tags
|
||||
|
||||
# Check if tag already exists
|
||||
if cleaned_tag_name not in [t.name for t in asset_data.tags]:
|
||||
try:
|
||||
asset_data.tags.new(cleaned_tag_name)
|
||||
print(f" + Added Asset Tag: '{cleaned_tag_name}'")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" Error adding tag '{cleaned_tag_name}': {e}")
|
||||
return False
|
||||
else:
|
||||
# print(f" Tag '{cleaned_tag_name}' already exists.") # Optional info
|
||||
return False # Not added because it existed
|
||||
|
||||
def get_supplier_tag_from_path(file_path_str, groupname):
|
||||
"""
|
||||
Determines supplier tag based on directory structure.
|
||||
Assumes structure is .../Supplier/Groupname/file.ext or .../Supplier/file.ext
|
||||
"""
|
||||
try:
|
||||
file_path = Path(file_path_str).resolve()
|
||||
groupname_lower = groupname.lower()
|
||||
|
||||
if not file_path.is_file():
|
||||
print(f" Warn (get_supplier_tag): Input path is not a file: {file_path_str}")
|
||||
return None
|
||||
|
||||
current_dir = file_path.parent # Directory containing the file
|
||||
if not current_dir:
|
||||
print(f" Warn (get_supplier_tag): Cannot get parent directory for {file_path_str}")
|
||||
return None # Cannot determine without parent
|
||||
|
||||
parent_dir = current_dir.parent # Directory potentially containing the 'supplier' name
|
||||
# Check if we are at the root or have no parent
|
||||
if not parent_dir or parent_dir == current_dir:
|
||||
# If the file is in the root scan directory or similar shallow path,
|
||||
# maybe the directory it's in IS the supplier tag? Or return None?
|
||||
# Returning current_dir.name might be unexpected, let's return None for safety.
|
||||
print(f" Warn (get_supplier_tag): File path too shallow to determine supplier reliably: {file_path_str}")
|
||||
return None
|
||||
|
||||
# Compare the file's directory name with the groupname
|
||||
if current_dir.name.lower() == groupname_lower:
|
||||
# Structure is likely .../Supplier/Groupname/file.ext
|
||||
# Return the name of the directory ABOVE the groupname directory
|
||||
return parent_dir.name
|
||||
else:
|
||||
# Structure is likely .../Supplier/file.ext
|
||||
# Return the name of the directory CONTAINING the file
|
||||
return current_dir.name
|
||||
|
||||
except Exception as e:
|
||||
print(f" Error getting supplier tag for {groupname} from path {file_path_str}: {e}")
|
||||
return None
|
||||
|
||||
def apply_asset_tags(parent_group, groupname, group_info):
|
||||
"""Applies various asset tags to the parent node group."""
|
||||
if not parent_group:
|
||||
return
|
||||
|
||||
# 1. Ensure group is marked as an asset
|
||||
try:
|
||||
if not parent_group.asset_data:
|
||||
print(f" Marking '{parent_group.name}' as asset for tagging.")
|
||||
parent_group.asset_mark()
|
||||
|
||||
# Ensure asset_data is available after marking
|
||||
if not parent_group.asset_data:
|
||||
print(f" Error: Could not access asset_data for '{parent_group.name}' after marking.")
|
||||
return
|
||||
|
||||
asset_data = parent_group.asset_data
|
||||
except Exception as e_mark:
|
||||
print(f" Error marking group '{parent_group.name}' as asset: {e_mark}")
|
||||
return # Cannot proceed without asset_data
|
||||
|
||||
# 2. Apply Supplier Tag (Current Requirement)
|
||||
try:
|
||||
# Find lowest resolution path (reuse logic from aspect ratio)
|
||||
lowest_res_path = None; found_res = False
|
||||
pbr_types_dict = group_info.get("pbr_types", {})
|
||||
# Check RESOLUTION_LABELS in order (assuming lowest is first)
|
||||
for res_label in RESOLUTION_LABELS:
|
||||
for res_data in pbr_types_dict.values(): # Check all PBR types for this res
|
||||
if res_label in res_data:
|
||||
lowest_res_path = res_data[res_label]
|
||||
found_res = True
|
||||
break # Found path for this resolution label
|
||||
if found_res:
|
||||
break # Found lowest available resolution path
|
||||
|
||||
if lowest_res_path:
|
||||
supplier_tag = get_supplier_tag_from_path(lowest_res_path, groupname)
|
||||
if supplier_tag:
|
||||
add_tag_if_new(asset_data, supplier_tag) # Use helper to add if new
|
||||
else:
|
||||
print(f" Warn (apply_asset_tags): No image path found for group '{groupname}' to determine supplier tag.")
|
||||
|
||||
except Exception as e_supp:
|
||||
print(f" Error during supplier tag processing for '{groupname}': {e_supp}")
|
||||
|
||||
# 3. --- Future Tagging Logic Placeholder ---
|
||||
# Example: Tag based on PBR Types present
|
||||
# try:
|
||||
# present_pbr_types = list(group_info.get("pbr_types", {}).keys())
|
||||
# for pbr_tag in present_pbr_types:
|
||||
# # Maybe add prefix or modify tag name
|
||||
# add_tag_if_new(asset_data, f"PBR_{pbr_tag}")
|
||||
# except Exception as e_pbr:
|
||||
# print(f" Error during PBR type tagging for '{groupname}': {e_pbr}")
|
||||
|
||||
# Example: Tag based on filename Tag (if not default like 'T-MR')
|
||||
# filename_tag = group_info.get("tag") # Need to store 'Tag' in group_info during scan
|
||||
# if filename_tag and filename_tag not in ["T-MR", "T-SG"]:
|
||||
# add_tag_if_new(asset_data, f"Tag_{filename_tag}")
|
||||
|
||||
# --- End Future Tagging Logic ---
|
||||
|
||||
|
||||
# --- Main Processing Function ---
|
||||
def process_textures_to_groups(root_directory):
|
||||
"""Scans textures, creates/updates node groups based on templates and manifest."""
|
||||
start_time = time.time()
|
||||
print(f"--- Starting Texture Processing ---")
|
||||
print(f"Scanning directory: {root_directory}")
|
||||
root_path = Path(root_directory)
|
||||
if not root_path.is_dir():
|
||||
print(f"Error: Directory not found: {root_directory}")
|
||||
return False # Indicate failure
|
||||
|
||||
# --- Manifest Setup ---
|
||||
current_blend_filepath = bpy.data.filepath
|
||||
manifest_path = get_manifest_path(current_blend_filepath)
|
||||
manifest_data = {}
|
||||
manifest_enabled = False
|
||||
if manifest_path:
|
||||
manifest_data = load_manifest(manifest_path)
|
||||
manifest_enabled = True
|
||||
# Flag will be True if any change requires saving the manifest
|
||||
manifest_needs_saving = False
|
||||
# --- End Manifest Setup ---
|
||||
|
||||
# --- Load Templates ---
|
||||
template_parent = bpy.data.node_groups.get(PARENT_TEMPLATE_NAME)
|
||||
template_child = bpy.data.node_groups.get(CHILD_TEMPLATE_NAME)
|
||||
if not template_parent:
|
||||
print(f"Error: Parent template '{PARENT_TEMPLATE_NAME}' not found.")
|
||||
return False
|
||||
if not template_child:
|
||||
print(f"Error: Child template '{CHILD_TEMPLATE_NAME}' not found.")
|
||||
return False
|
||||
print(f"Found templates: '{PARENT_TEMPLATE_NAME}', '{CHILD_TEMPLATE_NAME}'")
|
||||
# --- End Load Templates ---
|
||||
|
||||
# --- Initialize Data Structures ---
|
||||
# Stores {"GroupName": {"pbr_types": {...}, "scaling": "...", "sg": False, "thumb": "..."}}
|
||||
texture_data = {}
|
||||
file_count, processed_files = 0, 0
|
||||
groups_created, groups_updated, child_groups_created, child_groups_updated = 0, 0, 0, 0
|
||||
nodes_updated, links_created = 0, 0
|
||||
# Cache for image datablocks loaded in THIS RUN only
|
||||
loaded_images_this_run = {}
|
||||
# --- End Initialize Data Structures ---
|
||||
|
||||
print("Scanning files...")
|
||||
# --- File Scanning ---
|
||||
for dirpath, _, filenames in os.walk(root_directory):
|
||||
for filename in filenames:
|
||||
file_path = Path(dirpath) / filename
|
||||
# Check extension
|
||||
if file_path.suffix.lower() not in VALID_EXTENSIONS:
|
||||
continue
|
||||
|
||||
file_count += 1
|
||||
filename_stem = file_path.stem
|
||||
parsed = parse_texture_filename(filename_stem)
|
||||
if not parsed:
|
||||
continue # Skip if filename doesn't match format
|
||||
|
||||
# Extract parts
|
||||
groupname = parsed["Groupname"]
|
||||
pbr_type = parsed["PBRType"]
|
||||
resolution_label = parsed["Resolution"].lower()
|
||||
scaling_str = parsed["Scaling"]
|
||||
tag_str = parsed["Tag"].upper()
|
||||
file_path_str = str(file_path)
|
||||
|
||||
# Validate resolution label
|
||||
if resolution_label not in RESOLUTION_LABELS:
|
||||
print(f"Warn: Skip '{filename}' - Invalid Res '{resolution_label}'. Expected one of {RESOLUTION_LABELS}")
|
||||
continue
|
||||
|
||||
# Ensure base structure for group exists in texture_data
|
||||
group_entry = texture_data.setdefault(groupname, {
|
||||
"pbr_types": {}, "scaling": None, "sg": False, "thumb": None
|
||||
})
|
||||
|
||||
# Store texture path under the specific PBR type and resolution
|
||||
group_entry["pbr_types"].setdefault(pbr_type, {})[resolution_label] = file_path_str
|
||||
|
||||
# Store scaling string ONCE per groupname (first encountered wins)
|
||||
if group_entry["scaling"] is None:
|
||||
group_entry["scaling"] = scaling_str
|
||||
elif group_entry["scaling"] != scaling_str:
|
||||
# Warn only once per group if inconsistency found
|
||||
if not group_entry.get("scaling_warning_printed", False):
|
||||
print(f" Warn: Inconsistent 'Scaling' string found for group '{groupname}'. "
|
||||
f"Using first encountered: '{group_entry['scaling']}'.")
|
||||
group_entry["scaling_warning_printed"] = True
|
||||
|
||||
# Track SG status and thumbnail path
|
||||
if tag_str == "T-SG":
|
||||
group_entry["sg"] = True
|
||||
# Use 1k COL-1 as the potential thumbnail source
|
||||
if resolution_label == "1k" and pbr_type == "COL-1":
|
||||
group_entry["thumb"] = file_path_str
|
||||
|
||||
processed_files += 1
|
||||
# --- End File Scanning ---
|
||||
|
||||
print(f"\nFile Scan Complete. Found {file_count} files, parsed {processed_files} valid textures.")
|
||||
total_groups_found = len(texture_data)
|
||||
print(f"Total unique Groupnames found: {total_groups_found}")
|
||||
if not texture_data:
|
||||
print("No valid textures found. Exiting.")
|
||||
return True # No work needed is considered success
|
||||
|
||||
print("\n--- Processing Node Groups ---")
|
||||
|
||||
all_groupnames = sorted(list(texture_data.keys()))
|
||||
processing_stopped_early = False
|
||||
|
||||
# --- Main Processing Loop ---
|
||||
for groupname in all_groupnames:
|
||||
group_info = texture_data[groupname] # Get pre-scanned info
|
||||
pbr_types_data = group_info.get("pbr_types", {})
|
||||
scaling_string_for_group = group_info.get("scaling")
|
||||
sg_status_for_group = group_info.get("sg", False)
|
||||
thumbnail_path_for_group = group_info.get("thumb")
|
||||
|
||||
target_parent_name = f"PBRSET_{groupname}"
|
||||
print(f"\nProcessing Group: '{target_parent_name}'")
|
||||
|
||||
parent_group = bpy.data.node_groups.get(target_parent_name)
|
||||
is_new_parent = False
|
||||
|
||||
# --- Find or Create Parent Group ---
|
||||
if parent_group is None:
|
||||
# Check batch limit BEFORE creating
|
||||
if groups_created >= MAX_NEW_GROUPS_PER_RUN:
|
||||
print(f"\n--- Reached NEW parent group limit ({MAX_NEW_GROUPS_PER_RUN}). Stopping. ---")
|
||||
processing_stopped_early = True
|
||||
break # Exit the main groupname loop
|
||||
|
||||
print(f" Creating new parent group: '{target_parent_name}'")
|
||||
parent_group = template_parent.copy()
|
||||
if not parent_group:
|
||||
print(f" Error: Failed copy parent template. Skip group '{groupname}'.")
|
||||
continue # Skip to next groupname
|
||||
parent_group.name = target_parent_name
|
||||
groups_created += 1
|
||||
is_new_parent = True
|
||||
|
||||
# --- Auto-Save Trigger ---
|
||||
# Trigger AFTER creating the group and incrementing counter
|
||||
if AUTO_SAVE_ENABLED and groups_created > 0 and groups_created % SAVE_INTERVAL == 0:
|
||||
if perform_safe_autosave(manifest_path, manifest_data):
|
||||
# If auto-save succeeded, manifest is up-to-date on disk
|
||||
manifest_needs_saving = False
|
||||
else:
|
||||
# Auto-save failed, continue but warn
|
||||
print("!!! WARNING: Auto-save failed. Continuing processing... !!!")
|
||||
# --- End Auto-Save Trigger ---
|
||||
|
||||
else: # Update Existing Parent Group
|
||||
print(f" Updating existing parent group: '{target_parent_name}'")
|
||||
groups_updated += 1
|
||||
# --- End Find or Create Parent Group ---
|
||||
|
||||
# --- Process Parent Group Internals ---
|
||||
# This block processes both newly created and existing parent groups
|
||||
try:
|
||||
# --- Calculate and Store Aspect Ratio Correction (Once per group) ---
|
||||
# Find the designated Value node in the parent template
|
||||
aspect_node_list = find_nodes_by_label(parent_group, ASPECT_RATIO_NODE_LABEL, 'VALUE')
|
||||
if aspect_node_list:
|
||||
aspect_node = aspect_node_list[0] # Assume first found is correct
|
||||
if scaling_string_for_group:
|
||||
# Find the path to the lowest resolution image available
|
||||
lowest_res_path = None; found_res = False
|
||||
# Check resolution labels in configured order (e.g., "1k", "2k"...)
|
||||
for res_label in RESOLUTION_LABELS:
|
||||
# Check all PBR types for this resolution
|
||||
for res_data in pbr_types_data.values():
|
||||
if res_label in res_data:
|
||||
lowest_res_path = res_data[res_label]
|
||||
found_res = True
|
||||
break # Found path for this resolution label
|
||||
if found_res:
|
||||
break # Found lowest available resolution path
|
||||
|
||||
if lowest_res_path:
|
||||
# Load the image (use cache if possible)
|
||||
img = None; img_load_error = False
|
||||
if lowest_res_path in loaded_images_this_run:
|
||||
img = loaded_images_this_run[lowest_res_path]
|
||||
img_load_error = (img is None) # Check if cached result was failure
|
||||
else:
|
||||
# Attempt to load if not cached
|
||||
try:
|
||||
img_path_obj = Path(lowest_res_path)
|
||||
if img_path_obj.is_file():
|
||||
img = bpy.data.images.load(lowest_res_path, check_existing=True)
|
||||
else:
|
||||
img_load_error = True
|
||||
print(f" Error: Aspect source image not found: {lowest_res_path}")
|
||||
if img is None and not img_load_error: # Check if load function returned None
|
||||
img_load_error = True
|
||||
print(f" Error: Failed loading aspect source image: {lowest_res_path}")
|
||||
except Exception as e_load_aspect:
|
||||
print(f" Error loading aspect source image: {e_load_aspect}")
|
||||
img_load_error = True
|
||||
# Cache the result (image object or None)
|
||||
loaded_images_this_run[lowest_res_path] = img if not img_load_error else None
|
||||
|
||||
if not img_load_error and img:
|
||||
# Get dimensions and calculate factor
|
||||
img_width, img_height = img.size[0], img.size[1]
|
||||
factor = calculate_aspect_ratio_factor(img_width, img_height, scaling_string_for_group)
|
||||
print(f" Calculated Aspect Ratio Factor: {factor:.4f} (from {img_width}x{img_height}, Scaling='{scaling_string_for_group}')")
|
||||
|
||||
# Store factor in node if value changed significantly
|
||||
if abs(aspect_node.outputs[0].default_value - factor) > 0.0001:
|
||||
aspect_node.outputs[0].default_value = factor
|
||||
print(f" Set '{ASPECT_RATIO_NODE_LABEL}' node value to {factor:.4f}")
|
||||
else:
|
||||
print(f" Warn: Could not load image '{lowest_res_path}' for aspect ratio calc.")
|
||||
else:
|
||||
print(f" Warn: No suitable image found (e.g., 1k) to calculate aspect ratio for '{groupname}'.")
|
||||
else:
|
||||
print(f" Warn: No Scaling string found for group '{groupname}'. Cannot calculate aspect ratio.")
|
||||
# else: # Optional Warning if node is missing from template
|
||||
# print(f" Warn: Value node '{ASPECT_RATIO_NODE_LABEL}' not found in parent group '{parent_group.name}'. Cannot store aspect ratio.")
|
||||
# --- End Aspect Ratio Correction ---
|
||||
|
||||
# Set SG Value
|
||||
sg_nodes = find_nodes_by_label(parent_group, SG_VALUE_NODE_LABEL, 'VALUE')
|
||||
if sg_nodes:
|
||||
sg_node = sg_nodes[0]
|
||||
target_val = 1.0 if sg_status_for_group else 0.0
|
||||
if abs(sg_node.outputs[0].default_value - target_val) > 0.001:
|
||||
sg_node.outputs[0].default_value = target_val
|
||||
print(f" Set '{SG_VALUE_NODE_LABEL}' to: {target_val}")
|
||||
|
||||
# Set Asset Info (Thumbnail Path Prop, Initial Preview & Tagging)
|
||||
# This block runs for both new and existing groups
|
||||
try:
|
||||
# 1. Set/Update Thumbnail Path Property & Mark Asset
|
||||
if not parent_group.asset_data:
|
||||
parent_group.asset_mark()
|
||||
print(f" Marked '{parent_group.name}' as asset.")
|
||||
# Update thumbnail property logic
|
||||
if thumbnail_path_for_group:
|
||||
thumb_path_obj = Path(thumbnail_path_for_group)
|
||||
if thumb_path_obj.is_file():
|
||||
if parent_group.get("thumbnail_filepath") != thumbnail_path_for_group:
|
||||
parent_group["thumbnail_filepath"] = thumbnail_path_for_group
|
||||
if not is_new_parent: print(f" Updated thumbnail path property.") # Log update only if not new
|
||||
elif "thumbnail_filepath" in parent_group:
|
||||
del parent_group["thumbnail_filepath"]
|
||||
if not is_new_parent: print(f" Removed thumbnail path property (file not found).")
|
||||
elif "thumbnail_filepath" in parent_group:
|
||||
del parent_group["thumbnail_filepath"]
|
||||
if not is_new_parent: print(f" Removed old thumbnail path property.")
|
||||
|
||||
# 2. Set Initial Preview (Only if NEW parent)
|
||||
if is_new_parent and thumbnail_path_for_group and Path(thumbnail_path_for_group).is_file():
|
||||
print(f" Attempting initial preview from '{Path(thumbnail_path_for_group).name}'...")
|
||||
try:
|
||||
with bpy.context.temp_override(id=parent_group):
|
||||
bpy.ops.ed.lib_id_load_custom_preview(filepath=thumbnail_path_for_group)
|
||||
print(f" Set initial custom preview.")
|
||||
except Exception as e_prev:
|
||||
print(f" Preview Error: {e_prev}")
|
||||
|
||||
# 3. Apply Asset Tags (Supplier, etc.)
|
||||
apply_asset_tags(parent_group, groupname, group_info)
|
||||
|
||||
except Exception as e_asset_info:
|
||||
print(f" Error setting asset info/tags: {e_asset_info}")
|
||||
# --- End Asset Info ---
|
||||
|
||||
|
||||
# --- Process Child Groups (PBR Types) ---
|
||||
for pbr_type, resolutions_data in pbr_types_data.items():
|
||||
# print(f" Processing PBR Type: {pbr_type}") # Can be verbose
|
||||
|
||||
# Find placeholder node in parent
|
||||
holder_nodes = find_nodes_by_label(parent_group, pbr_type, 'GROUP')
|
||||
if not holder_nodes:
|
||||
print(f" Warn: No placeholder node labeled '{pbr_type}' in parent group '{parent_group.name}'. Skipping PBR Type.")
|
||||
continue
|
||||
holder_node = holder_nodes[0] # Assume first is correct
|
||||
|
||||
# Determine child group name (Base64 encoded)
|
||||
logical_child_name = f"{groupname}_{pbr_type}"
|
||||
target_child_name_b64 = encode_name_b64(logical_child_name)
|
||||
|
||||
# Find or Create Child Group
|
||||
child_group = bpy.data.node_groups.get(target_child_name_b64)
|
||||
if child_group is None:
|
||||
# print(f" Creating new child group for '{pbr_type}'") # Verbose
|
||||
child_group = template_child.copy()
|
||||
if not child_group:
|
||||
print(f" Error: Failed copy child template. Skip PBR Type.")
|
||||
continue
|
||||
child_group.name = target_child_name_b64
|
||||
child_groups_created += 1
|
||||
else:
|
||||
# print(f" Updating existing child group for '{pbr_type}'") # Verbose
|
||||
child_groups_updated += 1
|
||||
|
||||
# Assign child group to placeholder if needed
|
||||
if holder_node.node_tree != child_group:
|
||||
holder_node.node_tree = child_group
|
||||
print(f" Assigned child group '{child_group.name}' to placeholder '{holder_node.label}'.")
|
||||
|
||||
# Connect placeholder output to parent output socket if needed
|
||||
try:
|
||||
source_socket = holder_node.outputs[0] if holder_node.outputs else None
|
||||
group_output_node = next((n for n in parent_group.nodes if n.type == 'GROUP_OUTPUT'), None)
|
||||
target_socket = None
|
||||
if group_output_node:
|
||||
target_socket = group_output_node.inputs.get(pbr_type) # Get socket by name/label
|
||||
|
||||
if source_socket and target_socket:
|
||||
# Check if link already exists
|
||||
link_exists = any(link.from_socket == source_socket and link.to_socket == target_socket for link in parent_group.links)
|
||||
if not link_exists:
|
||||
parent_group.links.new(source_socket, target_socket)
|
||||
links_created += 1
|
||||
print(f" Connected '{holder_node.label}' output to parent output socket '{pbr_type}'.")
|
||||
# else: # Optional warning if sockets aren't found
|
||||
# if not source_socket: print(f" Warn: No output socket found on placeholder '{holder_node.label}'.")
|
||||
# if not target_socket: print(f" Warn: No input socket '{pbr_type}' found on parent output node.")
|
||||
|
||||
except Exception as e_link:
|
||||
print(f" Error linking sockets for '{pbr_type}': {e_link}")
|
||||
|
||||
# Ensure parent output socket type is Color
|
||||
try:
|
||||
item = parent_group.interface.items_tree.get(pbr_type)
|
||||
if item and item.in_out == 'OUTPUT' and item.socket_type != 'NodeSocketColor':
|
||||
item.socket_type = 'NodeSocketColor'
|
||||
# print(f" Set parent output socket '{pbr_type}' type to Color.") # Optional info
|
||||
except Exception as e_sock:
|
||||
print(f" Error updating socket type for '{pbr_type}': {e_sock}")
|
||||
|
||||
|
||||
# --- Process Resolutions within Child Group ---
|
||||
for resolution_label, image_path_str in resolutions_data.items():
|
||||
|
||||
# Find image texture nodes within the CHILD group
|
||||
image_nodes = find_nodes_by_label(child_group, resolution_label, 'TEX_IMAGE')
|
||||
if not image_nodes:
|
||||
# print(f" Warn: No node labeled '{resolution_label}' found in child group for '{pbr_type}'.") # Optional
|
||||
continue
|
||||
|
||||
# --- >>> Manifest Check <<< ---
|
||||
is_processed = False
|
||||
if manifest_enabled: # Only check if manifest is enabled
|
||||
# Check if this specific group/pbr/res combo is done
|
||||
processed_resolutions = manifest_data.get(groupname, {}).get(pbr_type, [])
|
||||
if resolution_label in processed_resolutions:
|
||||
is_processed = True
|
||||
# print(f" Skipping {groupname}/{pbr_type}/{resolution_label} (Manifest)") # Verbose skip log
|
||||
|
||||
if is_processed:
|
||||
continue # Skip to the next resolution
|
||||
# --- >>> End Manifest Check <<< ---
|
||||
|
||||
# --- Load Image & Assign (if not skipped) ---
|
||||
# print(f" Processing Resolution: {resolution_label} for {pbr_type}") # Verbose
|
||||
img = None
|
||||
image_load_failed = False
|
||||
|
||||
# Check intra-run cache first
|
||||
if image_path_str in loaded_images_this_run:
|
||||
img = loaded_images_this_run[image_path_str]
|
||||
image_load_failed = (img is None) # Respect cached failure
|
||||
else:
|
||||
# Not cached in this run, attempt to load
|
||||
try:
|
||||
image_path = Path(image_path_str)
|
||||
if not image_path.is_file():
|
||||
print(f" Error: Image file not found: {image_path_str}")
|
||||
image_load_failed = True
|
||||
else:
|
||||
# Use check_existing=True to potentially reuse existing datablocks
|
||||
img = bpy.data.images.load(str(image_path), check_existing=True)
|
||||
if not img:
|
||||
print(f" Error: Failed loading image via bpy.data.images.load: {image_path_str}")
|
||||
image_load_failed = True
|
||||
# else: # Success block is handled below
|
||||
# pass
|
||||
except RuntimeError as e_runtime_load:
|
||||
print(f" Runtime Error loading image '{image_path_str}': {e_runtime_load}")
|
||||
image_load_failed = True
|
||||
except Exception as e_gen_load:
|
||||
print(f" Unexpected error loading image '{image_path_str}': {e_gen_load}")
|
||||
image_load_failed = True
|
||||
# Cache result (image object or None for failure)
|
||||
loaded_images_this_run[image_path_str] = img if not image_load_failed else None
|
||||
|
||||
# --- Process image if loaded/cached successfully ---
|
||||
if not image_load_failed and img:
|
||||
try:
|
||||
# Set Color Space
|
||||
correct_color_space = PBR_COLOR_SPACE_MAP.get(pbr_type, DEFAULT_COLOR_SPACE)
|
||||
if img.colorspace_settings.name != correct_color_space:
|
||||
print(f" Setting '{Path(img.filepath).name}' color space -> {correct_color_space}")
|
||||
img.colorspace_settings.name = correct_color_space
|
||||
|
||||
# Histogram Stats Calculation
|
||||
if resolution_label == "1k" and pbr_type in ["ROUGH", "DISP"]:
|
||||
target_node_label = f"{HISTOGRAM_NODE_PREFIX}{pbr_type}"
|
||||
target_nodes = find_nodes_by_label(parent_group, target_node_label, 'COMBXYZ')
|
||||
if target_nodes:
|
||||
target_node = target_nodes[0]
|
||||
try:
|
||||
socket_x = target_node.inputs.get("X")
|
||||
socket_y = target_node.inputs.get("Y")
|
||||
socket_z = target_node.inputs.get("Z")
|
||||
if socket_x and socket_y and socket_z:
|
||||
print(f" Calculating histogram stats for {pbr_type} 1K...")
|
||||
stats = calculate_image_stats(img)
|
||||
if stats:
|
||||
min_val, max_val, median_val = stats
|
||||
print(f" Stats: Min={min_val:.4f}, Max={max_val:.4f}, Median={median_val:.4f}")
|
||||
# Store stats in the Combine XYZ node
|
||||
socket_x.default_value = min_val
|
||||
socket_y.default_value = max_val
|
||||
socket_z.default_value = median_val
|
||||
print(f" Stored stats in '{target_node_label}'.")
|
||||
else:
|
||||
print(f" Warn: Failed calc stats for '{Path(img.filepath).name}'.")
|
||||
# else: print(f" Warn: Node '{target_node_label}' missing X/Y/Z sockets.")
|
||||
except Exception as e_combxyz_store:
|
||||
print(f" Error processing stats in '{target_node_label}': {e_combxyz_store}")
|
||||
# else: print(f" Warn: No stats node '{target_node_label}' found.")
|
||||
|
||||
# Assign Image to nodes in child group
|
||||
nodes_updated_this_res = 0
|
||||
for image_node in image_nodes:
|
||||
if image_node.image != img:
|
||||
image_node.image = img
|
||||
nodes_updated_this_res += 1
|
||||
nodes_updated += nodes_updated_this_res
|
||||
if nodes_updated_this_res > 0:
|
||||
print(f" Assigned image '{Path(img.filepath).name}' to {nodes_updated_this_res} node(s).")
|
||||
|
||||
# --- >>> Update Manifest <<< ---
|
||||
if manifest_enabled:
|
||||
# Ensure nested structure exists
|
||||
manifest_data.setdefault(groupname, {}).setdefault(pbr_type, [])
|
||||
# Add resolution if not already present
|
||||
if resolution_label not in manifest_data[groupname][pbr_type]:
|
||||
manifest_data[groupname][pbr_type].append(resolution_label)
|
||||
# Keep the list sorted for consistency in the JSON file
|
||||
manifest_data[groupname][pbr_type].sort()
|
||||
manifest_needs_saving = True # Mark that we need to save later
|
||||
# print(f" Marked {groupname}/{pbr_type}/{resolution_label} processed in manifest.") # Verbose
|
||||
# --- >>> End Update Manifest <<< ---
|
||||
|
||||
except Exception as e_proc_img:
|
||||
print(f" Error during post-load processing for image '{image_path_str}': {e_proc_img}")
|
||||
# Continue to next resolution even if post-load fails
|
||||
# --- End Process image ---
|
||||
# --- End Resolution Loop ---
|
||||
# --- End PBR Type Loop ---
|
||||
except Exception as e_group:
|
||||
print(f" !!! ERROR processing group '{groupname}': {e_group} !!!")
|
||||
import traceback; traceback.print_exc()
|
||||
continue # Continue to next groupname
|
||||
|
||||
# --- End Main Processing Loop ---
|
||||
|
||||
# --- Final Manifest Save ---
|
||||
# Save if manifest is enabled AND changes were made since the last save/start.
|
||||
# This happens even if the script stopped early due to MAX_NEW_GROUPS_PER_RUN.
|
||||
if manifest_enabled and manifest_needs_saving:
|
||||
print("\n--- Attempting Final Manifest Save (End of Run) ---")
|
||||
if save_manifest(manifest_path, manifest_data):
|
||||
print(" Manifest saved successfully.")
|
||||
# Error message handled within save_manifest
|
||||
# --- End Final Manifest Save ---
|
||||
|
||||
# --- Final Summary ---
|
||||
end_time = time.time(); duration = end_time - start_time
|
||||
print("\n--- Script Run Finished ---")
|
||||
if processing_stopped_early:
|
||||
print(f"--- NOTE: Reached NEW parent group processing limit ({MAX_NEW_GROUPS_PER_RUN}). ---")
|
||||
print(f"--- You may need to SAVE manually, REVERT/RELOAD file, and RUN SCRIPT AGAIN. ---")
|
||||
print(f"Duration: {duration:.2f} seconds this run.")
|
||||
print(f"Summary: New Parents={groups_created}, Updated Parents={groups_updated}, New Children={child_groups_created}, Updated Children={child_groups_updated}.")
|
||||
print(f" Images assigned={nodes_updated} times. Links created={links_created}.")
|
||||
# Add other stats if needed, e.g., number of tags added
|
||||
# --- End Final Summary ---
|
||||
|
||||
return True # Indicate successful completion (or reaching limit)
|
||||
|
||||
|
||||
# --- How to Run ---
|
||||
# 1. Ensure 'numpy' is available in Blender's Python environment.
|
||||
# 2. Create Node Group "Template_PBRSET": Configure placeholders, Value nodes (SG, Aspect Ratio), Stats nodes, outputs.
|
||||
# 3. Create Node Group "Template_PBRTYPE": Configure Image Texture nodes labeled by resolution.
|
||||
# 4. !! SAVE YOUR BLEND FILE AT LEAST ONCE !! for manifest, auto-saving, and auto-reloading to work.
|
||||
# 5. Adjust variables in the '--- USER CONFIGURATION ---' section at the top as needed.
|
||||
# 6. Paste into Blender's Text Editor and run (Alt+P or Run Script button). Check Window -> Toggle System Console.
|
||||
# 7. If script stops due to limit: SAVE manually, REVERT/REOPEN file, RUN SCRIPT AGAIN. Manifest prevents reprocessing.
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(f"Script execution started at: {time.strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
|
||||
# Pre-run Checks using variables from CONFIG section
|
||||
valid_run_setup = True
|
||||
try:
|
||||
tex_dir_path = Path(texture_root_directory)
|
||||
# Basic check if path looks like a placeholder or doesn't exist
|
||||
if texture_root_directory == r"C:\path\to\your\texture\library" or not tex_dir_path.is_dir() :
|
||||
print(f"\nERROR: 'texture_root_directory' is invalid or a placeholder.")
|
||||
print(f" Current value: '{texture_root_directory}'")
|
||||
valid_run_setup = False
|
||||
except Exception as e_path:
|
||||
print(f"\nERROR checking texture_root_directory: {e_path}")
|
||||
valid_run_setup = False
|
||||
|
||||
# Check templates
|
||||
if not bpy.data.node_groups.get(PARENT_TEMPLATE_NAME):
|
||||
print(f"\nERROR: Parent template node group '{PARENT_TEMPLATE_NAME}' not found.")
|
||||
valid_run_setup = False
|
||||
if not bpy.data.node_groups.get(CHILD_TEMPLATE_NAME):
|
||||
print(f"\nERROR: Child template node group '{CHILD_TEMPLATE_NAME}' not found.")
|
||||
valid_run_setup = False
|
||||
|
||||
# Check numpy (needed for stats)
|
||||
try:
|
||||
import numpy
|
||||
except ImportError:
|
||||
print("\nCRITICAL ERROR: Python library 'numpy' not found (required for image stats).")
|
||||
print(" Please install numpy into Blender's Python environment.")
|
||||
valid_run_setup = False
|
||||
|
||||
# Execute main function if setup checks pass
|
||||
script_completed_successfully = False
|
||||
if valid_run_setup:
|
||||
# Check if file is saved before running features that depend on it
|
||||
if not bpy.data.filepath:
|
||||
print("\nWARNING: Blend file not saved. Manifest, Auto-Save, and Auto-Reload features disabled.")
|
||||
script_completed_successfully = process_textures_to_groups(texture_root_directory)
|
||||
else:
|
||||
print("\nScript aborted due to configuration errors.")
|
||||
|
||||
# --- Final Save & Reload ---
|
||||
# Use config variables directly as they are in module scope
|
||||
if script_completed_successfully and AUTO_RELOAD_ON_FINISH:
|
||||
if bpy.data.filepath: # Only if file is saved
|
||||
print("\n--- Auto-saving and reloading blend file ---")
|
||||
try:
|
||||
bpy.ops.wm.save_mainfile()
|
||||
print(" Blend file saved.")
|
||||
print(" Reloading...")
|
||||
# Ensure script execution stops cleanly before reload starts
|
||||
bpy.ops.wm.open_mainfile(filepath=bpy.data.filepath)
|
||||
# Script execution effectively stops here upon reload
|
||||
except Exception as e:
|
||||
print(f"!!! ERROR during final save/reload: {e} !!!")
|
||||
else:
|
||||
print("\nSkipping final save & reload because the blend file is not saved.")
|
||||
# --- End Final Save & Reload ---
|
||||
|
||||
# This print might not be reached if reload occurs
|
||||
print(f"Script execution finished processing at: {time.strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,90 +0,0 @@
|
||||
# Final Plan for Updating documentation.txt with Debugging Details
|
||||
|
||||
This document outlines the final plan for enhancing `documentation.txt` with detailed internal information crucial for debugging the Asset Processor Tool. This plan incorporates analysis of `asset_processor.py`, `configuration.py`, `main.py`, and `gui/processing_handler.py`.
|
||||
|
||||
## Task Objective
|
||||
|
||||
Analyze relevant source code (`asset_processor.py`, `configuration.py`, `main.py`, `gui/processing_handler.py`) to gather specific details about internal logic, state management, error handling, data structures, concurrency, resource management, and limitations. Integrate this information into the existing `documentation.txt` to aid developers in debugging.
|
||||
|
||||
## Analysis Steps Completed
|
||||
|
||||
1. Read and analyzed `readme.md`.
|
||||
2. Listed code definitions in the root directory (`.`).
|
||||
3. Listed code definitions in the `gui/` directory.
|
||||
4. Read and analyzed `asset_processor.py`.
|
||||
5. Read and analyzed `configuration.py`.
|
||||
6. Read and analyzed `main.py`.
|
||||
7. Read and analyzed `gui/processing_handler.py`.
|
||||
|
||||
## Final Integration Plan (Merged Structure)
|
||||
|
||||
1. **Add New Top-Level Section:**
|
||||
* Append the following section header to the end of `documentation.txt`:
|
||||
```
|
||||
================================
|
||||
Internal Details for Debugging
|
||||
================================
|
||||
```
|
||||
|
||||
2. **Create Subsections:**
|
||||
* Under the new "Internal Details" section, create the following subsections (renumbering from 7 onwards):
|
||||
* `7. Internal Logic & Algorithms`
|
||||
* `8. State Management`
|
||||
* `9. Error Handling & Propagation`
|
||||
* `10. Key Data Structures`
|
||||
* `11. Concurrency Models (CLI & GUI)`
|
||||
* `12. Resource Management`
|
||||
* `13. Known Limitations & Edge Cases`
|
||||
|
||||
3. **Populate Subsections with Specific Details:**
|
||||
|
||||
* **7. Internal Logic & Algorithms:**
|
||||
* **Configuration Preparation (`Configuration` class):** Detail the `__init__` process: loading `config.py` and preset JSON, validating structure (`_validate_configs`), compiling regex (`_compile_regex_patterns` using `_fnmatch_to_regex`). Mention compiled patterns storage.
|
||||
* **CLI Argument Parsing (`main.py:setup_arg_parser`):** Briefly describe `argparse` usage and key flags influencing execution.
|
||||
* **Output Directory Resolution (`main.py:main`):** Explain how the final output path is determined and resolved.
|
||||
* **Asset Processing (`AssetProcessor` class):**
|
||||
* *Classification (`_inventory_and_classify_files`):* Describe multi-pass approach using compiled regex from `Configuration`. Detail variant sorting criteria.
|
||||
* *Map Processing (`_process_maps`):* Detail image loading, Gloss->Rough inversion, resizing, format determination (using `Configuration` rules), bit depth conversion, stats calculation, aspect ratio logic, save fallback.
|
||||
* *Merging (`_merge_maps`):* Detail common resolution finding, input loading, channel merging (using `Configuration` rules), output determination, save fallback.
|
||||
* *Metadata (`_determine_base_metadata`, etc.):* Summarize base name extraction, category/archetype determination (using `Configuration` rules), `metadata.json` population.
|
||||
* **Blender Integration (`main.py:run_blender_script`, `gui/processing_handler.py:_run_blender_script_subprocess`):** Explain subprocess execution, command construction (`-b`, `--python`, `--`), argument passing (`asset_root_dir`).
|
||||
|
||||
* **8. State Management:**
|
||||
* `Configuration` object: Holds loaded config state and compiled regex. Instantiated per worker.
|
||||
* `AssetProcessor`: Primarily stateless between `process()` calls. Internal state within `process()` managed by local variables (e.g., `current_asset_metadata`). `self.classified_files` populated once and filtered per asset.
|
||||
* `main.py`: Tracks overall CLI run counts (processed, skipped, failed).
|
||||
* `gui/processing_handler.py`: Manages GUI processing run state via flags (`_is_running`, `_cancel_requested`) and stores `Future` objects (`_futures`).
|
||||
|
||||
* **9. Error Handling & Propagation:**
|
||||
* Custom Exceptions: `AssetProcessingError`, `ConfigurationError`.
|
||||
* `Configuration`: Raises `ConfigurationError` on load/validation failure. Logs regex compilation warnings.
|
||||
* `AssetProcessor`: Catches exceptions within per-asset loop, logs error, marks asset "failed", continues loop. Handles specific save fallbacks (EXR->PNG). Raises `AssetProcessingError` for critical setup failures.
|
||||
* Worker Wrapper (`main.py:process_single_asset_wrapper`): Catches exceptions from `Configuration`/`AssetProcessor`, logs, returns "failed" status tuple.
|
||||
* Process Pool (`main.py`, `gui/processing_handler.py`): `try...except` around executor block catches pool-level errors.
|
||||
* GUI Communication (`ProcessingHandler`): Catches errors during future result retrieval, emits failure status via signals.
|
||||
* Blender Scripts: Checks subprocess return code, logs stderr. Catches `FileNotFoundError`.
|
||||
|
||||
* **10. Key Data Structures:**
|
||||
* `Configuration` attributes: `compiled_map_keyword_regex`, `compiled_extra_regex`, etc. (compiled `re.Pattern` objects).
|
||||
* `AssetProcessor` structures: `self.classified_files` (dict[str, list[dict]]), `processed_maps_details_asset` (dict[str, dict[str, dict]]), `file_to_base_name_map` (dict[Path, Optional[str]]).
|
||||
* Return values: Status dictionary from `AssetProcessor.process()`, status tuple from `process_single_asset_wrapper`.
|
||||
* `ProcessingHandler._futures`: dict[Future, str].
|
||||
|
||||
* **11. Concurrency Models (CLI & GUI):**
|
||||
* **Common Core:** Both use `concurrent.futures.ProcessPoolExecutor` running `main.process_single_asset_wrapper`. `Configuration` and `AssetProcessor` instantiated within each worker process for isolation.
|
||||
* **CLI Orchestration (`main.py:run_processing`):** Direct executor usage, `as_completed` gathers results synchronously.
|
||||
* **GUI Orchestration (`gui/processing_handler.py`):** `ProcessingHandler` (QObject) runs executor logic in a `QThread`. Results processed in handler thread, communicated asynchronously to UI thread via Qt signals (`progress_updated`, `file_status_updated`, `processing_finished`).
|
||||
* **Cancellation (`gui/processing_handler.py:request_cancel`):** Sets flag, attempts `executor.shutdown(wait=False)`, tries cancelling pending futures. Limitation: Does not stop already running workers.
|
||||
|
||||
* **12. Resource Management:**
|
||||
* `Configuration`: Uses `with open` for preset files.
|
||||
* `AssetProcessor`: Manages temporary workspace (`tempfile.mkdtemp`, `shutil.rmtree` in `finally`). Uses `with open` for metadata JSON.
|
||||
* `ProcessPoolExecutor`: Lifecycle managed via `with` statement in `main.py` and `gui/processing_handler.py`.
|
||||
|
||||
* **13. Known Limitations & Edge Cases:**
|
||||
* Configuration: Basic structural validation only; regex compilation errors are warnings; `_fnmatch_to_regex` helper is basic.
|
||||
* AssetProcessor: Relies heavily on filename patterns; high memory potential for large images; limited intra-asset error recovery; simplified prediction logic.
|
||||
* CLI: Basic preset file existence check only before starting workers; Blender executable finding logic order (config > PATH).
|
||||
* GUI Concurrency: Cancellation doesn't stop currently executing worker processes.
|
||||
|
||||
4. **Switch Mode:** Request switching to Code mode to apply these changes by appending to `documentation.txt`.
|
||||
@ -1,269 +0,0 @@
|
||||
================================
|
||||
Asset Processor Tool - Developer Documentation
|
||||
================================
|
||||
|
||||
This document provides a concise overview of the Asset Processor Tool's codebase for developers joining the project. It focuses on the architecture, key components, and development workflow.
|
||||
|
||||
**NOTE:** This documentation strictly excludes details on environment setup, dependency installation, building the project, or deployment procedures. It assumes familiarity with Python and the relevant libraries (OpenCV, NumPy, PySide6).
|
||||
|
||||
--------------------------------
|
||||
1. Project Overview
|
||||
--------------------------------
|
||||
|
||||
* **Purpose:** To process 3D asset source files (texture sets, models, etc., typically from ZIP archives or folders) into a standardized library format.
|
||||
* **Core Functionality:** Uses configurable JSON presets to interpret different asset sources, automating tasks like file classification, image resizing, channel merging, and metadata generation.
|
||||
* **High-Level Architecture:** Consists of a core processing engine (`AssetProcessor`), a configuration system handling presets (`Configuration`), multiple interfaces (GUI, CLI, Directory Monitor), and optional integration with Blender for automated material/nodegroup creation.
|
||||
|
||||
--------------------------------
|
||||
2. Codebase Structure
|
||||
--------------------------------
|
||||
|
||||
Key files and directories:
|
||||
|
||||
* `asset_processor.py`: Contains the `AssetProcessor` class, the core logic for processing a single asset through the pipeline. Includes methods for classification, map processing, merging, metadata generation, and output organization. Also provides methods for predicting output structure used by the GUI.
|
||||
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config.py` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency.
|
||||
* `config.py`: Stores global default settings, constants, and core rules (e.g., standard map types, default resolutions, merge rules, output format rules, Blender paths).
|
||||
* `main.py`: Entry point for the Command-Line Interface (CLI). Handles argument parsing, logging setup, parallel processing orchestration (using `concurrent.futures.ProcessPoolExecutor`), calls `AssetProcessor` via a wrapper function, and optionally triggers Blender scripts.
|
||||
* `monitor.py`: Implements the automated directory monitoring feature using the `watchdog` library. Contains the `ZipHandler` class to detect new ZIP files and trigger processing via `main.run_processing`.
|
||||
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6.
|
||||
* `main_window.py`: Defines the `MainWindow` class, the main application window structure, UI layout (preset editor, processing panel, drag-and-drop, preview table, controls), event handling (button clicks, drag/drop), and menu setup. Manages GUI-specific logging (`QtLogHandler`).
|
||||
* `processing_handler.py`: Defines the `ProcessingHandler` class (runs on a `QThread`). Manages the execution of the main asset processing pipeline (using `ProcessPoolExecutor`) and Blender script execution in the background to keep the GUI responsive. Communicates progress and results back to the `MainWindow` via signals.
|
||||
* `prediction_handler.py`: Defines the `PredictionHandler` class (runs on a `QThread`). Manages background file analysis/preview generation by calling `AssetProcessor.get_detailed_file_predictions()`. Sends results back to the `MainWindow` via signals to update the preview table.
|
||||
* `preview_table_model.py`: Defines `PreviewTableModel` (inherits `QAbstractTableModel`) and `PreviewSortFilterProxyModel` for managing and displaying data in the GUI's preview table, including custom sorting logic.
|
||||
* `blenderscripts/`: Contains Python scripts (`create_nodegroups.py`, `create_materials.py`) designed to be executed *within* Blender, typically triggered by the main tool after processing to automate PBR nodegroup and material setup in `.blend` files.
|
||||
* `Presets/`: Contains supplier-specific configuration files in JSON format (e.g., `Poliigon.json`). These define rules for interpreting asset filenames, classifying maps, handling variants, etc. `_template.json` serves as a base for new presets.
|
||||
* `Testfiles/`: Contains example input assets for testing purposes.
|
||||
* `Tickets/`: Directory for issue and feature tracking using Markdown files.
|
||||
|
||||
--------------------------------
|
||||
3. Key Components/Modules
|
||||
--------------------------------
|
||||
|
||||
* **`AssetProcessor` (`asset_processor.py`):** The heart of the tool. Orchestrates the entire processing pipeline for a single input asset (ZIP or folder). Responsibilities include workspace management, file classification, metadata extraction, map processing (resizing, format conversion), channel merging, `metadata.json` generation, and organizing final output files.
|
||||
* **`Configuration` (`configuration.py`):** Manages the loading and merging of configuration settings. Takes a preset name, loads defaults from `config.py`, loads the specified `Presets/*.json`, merges them, validates settings, and pre-compiles regex patterns defined in the preset for efficient use by `AssetProcessor`.
|
||||
* **`MainWindow` (`gui/main_window.py`):** The main class for the GUI application. Sets up the UI layout, connects user actions (button clicks, drag/drop) to slots, manages the preset editor, interacts with background handlers (`ProcessingHandler`, `PredictionHandler`) via signals/slots, and displays feedback (logs, progress, status).
|
||||
* **`ProcessingHandler` (`gui/processing_handler.py`):** Handles the execution of the core asset processing logic and Blender scripts in a background thread for the GUI. Manages the `ProcessPoolExecutor` for parallel asset processing and communicates progress/results back to the `MainWindow`.
|
||||
* **`PredictionHandler` (`gui/prediction_handler.py`):** Handles the generation of file classification previews in a background thread for the GUI. Calls `AssetProcessor`'s prediction methods and sends results back to the `MainWindow` to populate the preview table without blocking the UI.
|
||||
* **`ZipHandler` (`monitor.py`):** A `watchdog` event handler used by `monitor.py`. Detects newly created ZIP files in the monitored input directory, validates the filename format (for preset extraction), and triggers the main processing logic via `main.run_processing`.
|
||||
|
||||
--------------------------------
|
||||
4. Core Concepts & Data Flow
|
||||
--------------------------------
|
||||
|
||||
* **Preset-Driven Configuration:**
|
||||
* Global defaults are set in `config.py`.
|
||||
* Supplier-specific rules (filename patterns, map keywords, variant handling, etc.) are defined using regex in `Presets/*.json` files.
|
||||
* The `Configuration` class loads `config.py` and merges it with the selected preset JSON, providing a unified configuration object to the `AssetProcessor`. Regex patterns are pre-compiled during `Configuration` initialization for performance.
|
||||
|
||||
* **Asset Processing Pipeline (Simplified Flow):**
|
||||
1. **Workspace Setup:** Create a temporary directory.
|
||||
2. **Extract/Copy:** Extract ZIP or copy folder contents to the workspace.
|
||||
3. **Classify Files:** Scan workspace, use compiled regex from `Configuration` to classify files (Map, Model, Extra, Ignored, Unrecognized). Handle 16-bit variants and assign suffixes based on rules.
|
||||
4. **Determine Metadata:** Extract asset name, category, archetype based on preset rules.
|
||||
5. **Skip Check:** If overwrite is false, check if output already exists; if so, skip this asset.
|
||||
6. **Process Maps:** Load images, resize (no upscale), convert format/bit depth based on complex rules (`config.py` and preset), handle Gloss->Roughness inversion, calculate stats, determine aspect ratio change. Save processed maps.
|
||||
7. **Merge Maps:** Combine channels from different processed maps based on `MAP_MERGE_RULES` in `config.py`. Save merged maps.
|
||||
8. **Generate `metadata.json`:** Collect all relevant information (map details, stats, aspect ratio, category, etc.) and write to `metadata.json` in the workspace.
|
||||
9. **Organize Output:** Create the final output directory structure (`<output_base>/<supplier>/<asset_name>/`) and move processed maps, merged maps, models, `metadata.json`, Extra files, and Ignored files into it.
|
||||
10. **Cleanup Workspace:** Delete the temporary directory.
|
||||
11. **(Optional) Blender Scripts:** If triggered via CLI/GUI, execute `blenderscripts/*.py` using the configured Blender executable via a subprocess.
|
||||
|
||||
* **Parallel Processing:**
|
||||
* Multiple input assets are processed concurrently using `concurrent.futures.ProcessPoolExecutor`.
|
||||
* This pool is managed by `main.py` (CLI) or `gui/processing_handler.py` (GUI).
|
||||
* Each asset runs in an isolated worker process, ensuring separate `Configuration` and `AssetProcessor` instances.
|
||||
|
||||
* **GUI Interaction & Threading:**
|
||||
* The GUI (`PySide6`) uses `QThread` to run `ProcessingHandler` (asset processing) and `PredictionHandler` (file preview generation) in the background, preventing the UI from freezing.
|
||||
* Communication between the main UI thread (`MainWindow`) and background threads relies on Qt's signals and slots mechanism for thread safety (e.g., updating progress, status messages, preview table data).
|
||||
* `PredictionHandler` calls `AssetProcessor` methods to get file classification details, which are then sent back to `MainWindow` to populate the `PreviewTableModel`.
|
||||
|
||||
* **Output (`metadata.json`):**
|
||||
* A key output file generated for each processed asset.
|
||||
* Contains structured data about the asset: map filenames, resolutions, formats, bit depths, merged map details, calculated image statistics, aspect ratio change info, asset category/archetype, source preset used, list of ignored source files, etc. This file is intended for use by downstream tools or scripts (like the Blender integration scripts).
|
||||
|
||||
--------------------------------
|
||||
5. Development Workflow
|
||||
--------------------------------
|
||||
|
||||
* **Modifying Core Processing Logic:** Changes to how assets are classified, maps are processed/resized/converted, channels are merged, or metadata is generated typically involve editing the `AssetProcessor` class in `asset_processor.py`.
|
||||
* **Changing Global Settings/Rules:** Adjustments to default output paths, standard resolutions, default format rules, map merge definitions, or Blender paths should be made in `config.py`.
|
||||
* **Adding/Modifying Supplier Rules:** To add support for a new asset source or change how an existing one is interpreted, create or edit the corresponding JSON file in the `Presets/` directory. Refer to `_template.json` and existing presets for structure. Focus on defining accurate regex patterns in `map_type_mapping`, `bit_depth_variants`, `model_patterns`, `source_naming_convention`, etc.
|
||||
* **Adjusting CLI Behavior:** Changes to command-line arguments, argument parsing, or the overall CLI workflow are handled in `main.py`.
|
||||
* **Modifying the GUI:** UI layout changes, adding new controls, altering event handling, or modifying background task management for the GUI involves working within the `gui/` directory, primarily `main_window.py`, `processing_handler.py`, and `prediction_handler.py`. UI elements are built using PySide6 widgets.
|
||||
* **Enhancing Blender Integration:** Improvements or changes to how nodegroups or materials are created in Blender require editing the Python scripts within the `blenderscripts/` directory. Consider how these scripts are invoked and what data they expect (primarily from `metadata.json` and command-line arguments passed via subprocess calls in `main.py` or `gui/processing_handler.py`).
|
||||
|
||||
--------------------------------
|
||||
6. Coding Conventions
|
||||
--------------------------------
|
||||
|
||||
* **Object-Oriented:** The codebase heavily utilizes classes (e.g., `AssetProcessor`, `Configuration`, `MainWindow`, various Handlers).
|
||||
* **Type Hinting:** Python type hints are used throughout the code for clarity and static analysis.
|
||||
* **Logging:** Standard Python `logging` module is used for logging messages at different levels (DEBUG, INFO, WARNING, ERROR). The GUI uses a custom `QtLogHandler` to display logs in the UI console.
|
||||
* **Error Handling:** Uses standard `try...except` blocks and defines some custom exceptions (e.g., `ConfigurationError`, `AssetProcessingError`).
|
||||
* **Parallelism:** Uses `concurrent.futures.ProcessPoolExecutor` for CPU-bound tasks (asset processing).
|
||||
* **GUI:** Uses `PySide6` (Qt for Python) with signals and slots for communication between UI elements and background threads (`QThread`).
|
||||
* **Configuration:** Relies on Python modules (`config.py`) for core settings and JSON files (`Presets/`) for specific rule sets.
|
||||
* **File Paths:** Uses `pathlib.Path` for handling file system paths.
|
||||
================================
|
||||
Internal Details for Debugging
|
||||
================================
|
||||
|
||||
This section provides deeper technical details about the internal workings, intended to aid in debugging unexpected behavior.
|
||||
|
||||
--------------------------------
|
||||
7. Internal Logic & Algorithms
|
||||
--------------------------------
|
||||
|
||||
* **Configuration Preparation (`Configuration` class in `configuration.py`):**
|
||||
* Instantiated per preset (`__init__`).
|
||||
* Loads core settings from `config.py` using `importlib.util`.
|
||||
* Loads specified preset from `presets/{preset_name}.json`.
|
||||
* Validates basic structure of loaded settings (`_validate_configs`), checking for required keys and basic types (e.g., `map_type_mapping` is a list of dicts).
|
||||
* Compiles regex patterns (`_compile_regex_patterns`) from preset rules (extra, model, bit depth, map keywords) using `re.compile` (mostly case-insensitive) and stores them on the instance (e.g., `self.compiled_map_keyword_regex`). Uses `_fnmatch_to_regex` helper for basic wildcard conversion.
|
||||
|
||||
* **CLI Argument Parsing (`main.py:setup_arg_parser`):**
|
||||
* Uses `argparse` to define and parse command-line arguments.
|
||||
* Key arguments influencing flow: `--preset` (required), `--output-dir` (optional override), `--workers` (concurrency), `--overwrite` (force reprocessing), `--verbose` (logging level), `--nodegroup-blend`, `--materials-blend`.
|
||||
* Calculates a default worker count based on `os.cpu_count()`.
|
||||
|
||||
* **Output Directory Resolution (`main.py:main`):**
|
||||
* Determines the base output directory by checking `--output-dir` argument first, then falling back to `OUTPUT_BASE_DIR` from `config.py`.
|
||||
* Resolves the path to an absolute path and ensures the directory exists (`Path.resolve()`, `Path.mkdir(parents=True, exist_ok=True)`).
|
||||
|
||||
* **Asset Processing (`AssetProcessor` class in `asset_processor.py`):**
|
||||
* **Classification (`_inventory_and_classify_files`):**
|
||||
* Multi-pass approach: Explicit Extra (regex) -> Models (regex) -> Potential Maps (keyword regex) -> Standalone 16-bit check (regex) -> Prioritize 16-bit variants -> Final Maps -> Remaining as Unrecognised (Extra).
|
||||
* Uses compiled regex patterns provided by the `Configuration` object passed during initialization.
|
||||
* Sorts potential map variants based on: 1. Preset rule index, 2. Keyword index within rule, 3. Alphabetical path. Suffixes (`-1`, `-2`) are assigned later per-asset based on this sort order and `RESPECT_VARIANT_MAP_TYPES`.
|
||||
* **Map Processing (`_process_maps`):**
|
||||
* Loads images using `cv2.imread` (flags: `IMREAD_UNCHANGED` or `IMREAD_GRAYSCALE`). Converts loaded 3-channel images from BGR to RGB for internal consistency (stats, merging).
|
||||
* **Saving Channel Order:** Before saving with `cv2.imwrite`, 3-channel images are conditionally converted back from RGB to BGR *only* if the target output format is *not* EXR (e.g., for PNG, JPG, TIF). This ensures correct channel order for standard formats while preserving RGB for EXR. (Fix for ISSUE-010).
|
||||
* Handles Gloss->Roughness inversion: Loads gloss, inverts using float math (`1.0 - img/norm`), stores as float32 with original dtype. Prioritizes gloss source if both gloss and native rough exist.
|
||||
* Resizes using `cv2.resize` (interpolation: `INTER_LANCZOS4` for downscale, `INTER_CUBIC` for potential same-size/upscale - though upscaling is generally avoided by checks).
|
||||
* Determines output format based on hierarchy: `FORCE_LOSSLESS_MAP_TYPES` > `RESOLUTION_THRESHOLD_FOR_JPG` > Input format priority (TIF/EXR often lead to lossless) > Configured defaults (`OUTPUT_FORMAT_16BIT_PRIMARY`, `OUTPUT_FORMAT_8BIT`).
|
||||
* Determines output bit depth based on `MAP_BIT_DEPTH_RULES` ('respect' vs 'force_8bit').
|
||||
* Converts dtype before saving (e.g., float to uint8/uint16 using scaling factors 255.0/65535.0).
|
||||
* Calculates stats (`_calculate_image_stats`) on normalized float64 data (in RGB space) for a specific resolution (`CALCULATE_STATS_RESOLUTION`).
|
||||
* Calculates aspect ratio string (`_normalize_aspect_ratio_change`) based on relative dimension changes.
|
||||
* Handles save fallback: If primary 16-bit format (e.g., EXR) fails, attempts fallback (e.g., PNG).
|
||||
* **Merging (`_merge_maps_from_source`):**
|
||||
* Identifies the required *source* files for merge inputs based on classified files.
|
||||
* Determines common resolutions based on available processed maps (as a proxy for size compatibility).
|
||||
* Loads required source maps for each common resolution using the `_load_and_transform_source` helper (utilizing the cache).
|
||||
* Converts loaded inputs to float32 (normalized 0-1).
|
||||
* Injects default values (from rule `defaults`) for missing channels.
|
||||
* Merges channels using `cv2.merge`.
|
||||
* Determines output bit depth based on rule (`force_16bit`, `respect_inputs`).
|
||||
* Determines output format based on complex rules (`config.py` and preset), considering the highest format among *source* inputs if not forced lossless or over JPG threshold. Handles JPG 16-bit conflict by forcing 8-bit.
|
||||
* Saves the merged image using the `_save_image` helper, including final data type/color space conversions and fallback logic (e.g., EXR->PNG).
|
||||
* **Metadata (`_determine_base_metadata`, `_determine_single_asset_metadata`, `_generate_metadata_file`):**
|
||||
* Base name determined using `source_naming` separator/index from `Configuration`, with fallback to common prefix or input name. Handles multiple assets within one input.
|
||||
* Category determined by model presence or `decal_keywords` from `Configuration`.
|
||||
* Archetype determined by matching keywords in `archetype_rules` (from `Configuration`) against file stems/base name.
|
||||
* Final `metadata.json` populated by accumulating results (map details, stats, features, etc.) during the per-asset processing loop.
|
||||
|
||||
* **Blender Integration (`main.py:run_blender_script`, `gui/processing_handler.py:_run_blender_script_subprocess`):**
|
||||
* Uses `subprocess.run` to execute Blender.
|
||||
* Command includes `-b` (background), the target `.blend` file, `--python` followed by the script path (`blenderscripts/*.py`), and `--` separator.
|
||||
* Arguments after `--` (currently just the `asset_root_dir`, and optionally the nodegroup blend path for the materials script) are passed to the Python script via `sys.argv`.
|
||||
* Uses `--factory-startup` in GUI handler. Checks return code and logs stdout/stderr.
|
||||
|
||||
--------------------------------
|
||||
8. State Management
|
||||
--------------------------------
|
||||
|
||||
* **`Configuration` Object:** Holds the loaded and merged configuration state (core + preset) and compiled regex patterns. Designed to be immutable after initialization. Instantiated once per worker process.
|
||||
* **`AssetProcessor` Instance:** Primarily stateless between calls to `process()`. State *within* a `process()` call is managed through local variables scoped to the overall call or the per-asset loop (e.g., `current_asset_metadata`, `processed_maps_details_asset`). `self.classified_files` is populated once by `_inventory_and_classify_files` early in `process()` and then used read-only (filtered copies) within the per-asset loop.
|
||||
* **`main.py` (CLI):** Tracks overall run progress (processed, skipped, failed counts) based on results returned from worker processes.
|
||||
* **`gui/processing_handler.py`:** Manages the state of a GUI processing run using internal flags (`_is_running`, `_cancel_requested`) and stores `Future` objects in `self._futures` dictionary while the pool is active.
|
||||
|
||||
--------------------------------
|
||||
9. Error Handling & Propagation
|
||||
--------------------------------
|
||||
|
||||
* **Custom Exceptions:** `ConfigurationError` (raised by `Configuration` on load/validation failure), `AssetProcessingError` (raised by `AssetProcessor` for various processing failures).
|
||||
* **Configuration:** `ConfigurationError` halts initialization. Regex compilation errors are logged as warnings but do not stop initialization.
|
||||
* **AssetProcessor:** Uses `try...except Exception` within key pipeline steps (`_process_maps`, `_merge_maps`, etc.) and within the per-asset loop in `process()`. Errors specific to one asset are logged (`log.error(exc_info=True)`), the asset is marked "failed" in the returned status dictionary, and the loop continues to the next asset. Critical setup errors (e.g., workspace creation) raise `AssetProcessingError`, halting the entire `process()` call. Includes specific save fallback logic (EXR->PNG) on `cv2.imwrite` failure for 16-bit formats.
|
||||
* **Worker Wrapper (`main.py:process_single_asset_wrapper`):** Catches `ConfigurationError`, `AssetProcessingError`, and general `Exception` during worker execution. Logs the error and returns a ("failed", error_message) status tuple to the main process.
|
||||
* **Process Pool (`main.py`, `gui/processing_handler.py`):** The `with ProcessPoolExecutor(...)` block handles pool setup/teardown. A `try...except` around `as_completed` or `future.result()` catches critical worker failures (e.g., process crash).
|
||||
* **GUI Communication (`ProcessingHandler`):** Catches exceptions during `future.result()` retrieval. Emits `file_status_updated` signal with "failed" status and error message. Emits `processing_finished` with final counts.
|
||||
* **Blender Scripts:** Checks `subprocess.run` return code. Logs stderr as ERROR if return code is non-zero, otherwise as WARNING. Catches `FileNotFoundError` if the Blender executable path is invalid.
|
||||
|
||||
--------------------------------
|
||||
10. Key Data Structures
|
||||
--------------------------------
|
||||
|
||||
* **`Configuration` Instance Attributes:**
|
||||
* `compiled_map_keyword_regex`: `dict[str, list[tuple[re.Pattern, str, int]]]` (Base type -> list of compiled regex tuples)
|
||||
* `compiled_extra_regex`, `compiled_model_regex`: `list[re.Pattern]`
|
||||
* `compiled_bit_depth_regex_map`: `dict[str, re.Pattern]` (Base type -> compiled regex)
|
||||
* **`AssetProcessor` Internal Structures (within `process()`):**
|
||||
* `self.classified_files`: `dict[str, list[dict]]` (Category -> list of file info dicts like `{'source_path': Path, 'map_type': str, ...}`)
|
||||
* `processed_maps_details_asset`, `merged_maps_details_asset`: `dict[str, dict[str, dict]]` (Map Type -> Resolution Key -> Details Dict `{'path': Path, 'width': int, ...}`)
|
||||
* `file_to_base_name_map`: `dict[Path, Optional[str]]` (Source relative path -> Determined asset base name or None)
|
||||
* `current_asset_metadata`: `dict` (Accumulates name, category, archetype, stats, map details per asset)
|
||||
* **Return Values:**
|
||||
* `AssetProcessor.process()`: `Dict[str, List[str]]` (e.g., `{"processed": [...], "skipped": [...], "failed": [...]}`)
|
||||
* `main.process_single_asset_wrapper()`: `Tuple[str, str, Optional[str]]` (input_path, status_string, error_message)
|
||||
* **`ProcessingHandler._futures`:** `dict[Future, str]` (Maps `concurrent.futures.Future` object to the input path string)
|
||||
* **Image Data:** `numpy.ndarray` (Handled by OpenCV).
|
||||
|
||||
--------------------------------
|
||||
11. Concurrency Models (CLI & GUI)
|
||||
--------------------------------
|
||||
|
||||
* **Common Core:** Both CLI and GUI utilize `concurrent.futures.ProcessPoolExecutor` for parallel processing. The target function executed by workers is `main.process_single_asset_wrapper`.
|
||||
* **Isolation:** Crucially, `Configuration` and `AssetProcessor` objects are instantiated *within* the `process_single_asset_wrapper` function, meaning each worker process gets its own independent configuration and processor instance based on the arguments passed. This prevents state conflicts between concurrent asset processing tasks. Data is passed between the main process and workers via pickling of arguments and return values.
|
||||
* **CLI Orchestration (`main.py:run_processing`):**
|
||||
* Creates the `ProcessPoolExecutor`.
|
||||
* Submits all `process_single_asset_wrapper` tasks.
|
||||
* Uses `concurrent.futures.as_completed` to iterate over finished futures as they complete, blocking until the next one is done.
|
||||
* Gathers results synchronously within the main script's execution flow.
|
||||
* **GUI Orchestration (`gui/processing_handler.py`):**
|
||||
* The `ProcessingHandler` object (a `QObject`) contains the `run_processing` method.
|
||||
* This method is intended to be run in a separate `QThread` (managed by `MainWindow`) to avoid blocking the main UI thread.
|
||||
* Inside `run_processing`, it creates and manages the `ProcessPoolExecutor`.
|
||||
* It uses `as_completed` similarly to the CLI to iterate over finished futures.
|
||||
* **Communication:** Instead of blocking the thread gathering results, it emits Qt signals (`progress_updated`, `file_status_updated`, `processing_finished`) from within the `as_completed` loop. These signals are connected to slots in `MainWindow` (running on the main UI thread), allowing for thread-safe updates to the GUI (progress bar, table status, status bar messages).
|
||||
* **Cancellation (GUI - `gui/processing_handler.py:request_cancel`):**
|
||||
* Sets an internal `_cancel_requested` flag.
|
||||
* Attempts `executor.shutdown(wait=False)` which prevents new tasks from starting and may cancel pending ones (depending on Python version).
|
||||
* Manually iterates through stored `_futures` and calls `future.cancel()` on those not yet running or done.
|
||||
* **Limitation:** This does *not* forcefully terminate worker processes that are already executing the `process_single_asset_wrapper` function. Cancellation primarily affects pending tasks and the processing of results from already running tasks (they will be marked as failed/cancelled when their future completes).
|
||||
|
||||
--------------------------------
|
||||
12. Resource Management
|
||||
--------------------------------
|
||||
|
||||
* **Configuration:** Preset JSON files are opened and closed using `with open(...)`.
|
||||
* **AssetProcessor:**
|
||||
* Temporary workspace directory created using `tempfile.mkdtemp()`.
|
||||
* Cleanup (`_cleanup_workspace`) uses `shutil.rmtree()` and is called within a `finally` block in the main `process()` method, ensuring cleanup attempt even if errors occur.
|
||||
* Metadata JSON file written using `with open(...)`.
|
||||
* Image data is loaded into memory using OpenCV/NumPy; memory usage depends on image size and number of concurrent workers.
|
||||
* **Process Pool:** The `ProcessPoolExecutor` manages the lifecycle of worker processes. Using it within a `with` statement (as done in `main.py` and `gui/processing_handler.py`) ensures proper shutdown and resource release for the pool itself.
|
||||
|
||||
--------------------------------
|
||||
13. Known Limitations & Edge Cases
|
||||
--------------------------------
|
||||
|
||||
* **Configuration:**
|
||||
* Validation (`_validate_configs`) is primarily structural (key presence, basic types), not deeply logical (e.g., doesn't check if regex patterns are *sensible*).
|
||||
* Regex compilation errors in `_compile_regex_patterns` are logged as warnings but don't prevent `Configuration` initialization, potentially leading to unexpected classification later.
|
||||
* `_fnmatch_to_regex` helper only handles basic `*` and `?` wildcards. Complex fnmatch patterns might not translate correctly.
|
||||
* **AssetProcessor:**
|
||||
* Heavily reliant on correct filename patterns and rules defined in presets. Ambiguous or incorrect patterns lead to misclassification.
|
||||
* Potential for high memory usage when processing very large images, especially with many workers.
|
||||
* Error handling within `process()` is per-asset; a failure during map processing for one asset marks the whole asset as failed, without attempting other maps for that asset. No partial recovery within an asset.
|
||||
* Gloss->Roughness inversion assumes gloss map is single channel or convertible to grayscale.
|
||||
* `predict_output_structure` and `get_detailed_file_predictions` use simplified logic (e.g., assuming PNG output, highest resolution only) and may not perfectly match final output names/formats in all cases.
|
||||
* Filename sanitization (`_sanitize_filename`) is basic and might not cover all edge cases for all filesystems.
|
||||
* **CLI (`main.py`):**
|
||||
* Preset existence check (`{preset}.json`) happens only in the main process before workers start.
|
||||
* Blender executable finding logic relies on `config.py` path being valid or `blender` being in the system PATH.
|
||||
* **GUI Concurrency (`gui/processing_handler.py`):**
|
||||
* Cancellation (`request_cancel`) is not immediate for tasks already running in worker processes. It prevents new tasks and stops processing results from completed futures once the flag is checked.
|
||||
* **General:**
|
||||
* Limited input format support (ZIP archives, folders). Internal file formats limited by OpenCV (`cv2.imread`, `cv2.imwrite`). Optional `OpenEXR` package recommended for full EXR support.
|
||||
* Error messages propagated from workers might lack full context in some edge cases.
|
||||
@ -1,70 +0,0 @@
|
||||
# Asset Processor Tool Documentation Plan
|
||||
|
||||
This document outlines the proposed structure for the documentation of the Asset Processor Tool, based on the content from `readme.md` and `documentation.txt`. The goal is to create a clear, modular, and comprehensive documentation set within a new `Documentation` directory.
|
||||
|
||||
## Proposed Directory Structure
|
||||
|
||||
```
|
||||
Documentation/
|
||||
├── 00_Overview.md
|
||||
├── 01_User_Guide/
|
||||
│ ├── 01_Introduction.md
|
||||
│ ├── 02_Features.md
|
||||
│ ├── 03_Installation.md
|
||||
│ ├── 04_Configuration_and_Presets.md
|
||||
│ ├── 05_Usage_GUI.md
|
||||
│ ├── 06_Usage_CLI.md
|
||||
│ ├── 07_Usage_Monitor.md
|
||||
│ ├── 08_Usage_Blender.md
|
||||
│ ├── 09_Output_Structure.md
|
||||
│ └── 10_Docker.md
|
||||
└── 02_Developer_Guide/
|
||||
├── 01_Architecture.md
|
||||
├── 02_Codebase_Structure.md
|
||||
├── 03_Key_Components.md
|
||||
├── 04_Configuration_System_and_Presets.md
|
||||
├── 05_Processing_Pipeline.md
|
||||
├── 06_GUI_Internals.md
|
||||
├── 07_Monitor_Internals.md
|
||||
├── 08_Blender_Integration_Internals.md
|
||||
├── 09_Development_Workflow.md
|
||||
├── 10_Coding_Conventions.md
|
||||
└── 11_Debugging_Notes.md
|
||||
```
|
||||
|
||||
## File Content Breakdown
|
||||
|
||||
### `Documentation/00_Overview.md`
|
||||
|
||||
* Project purpose, scope, and intended audience.
|
||||
* High-level summary of the tool's functionality.
|
||||
* Table of Contents for the entire documentation set.
|
||||
|
||||
### `Documentation/01_User_Guide/`
|
||||
|
||||
* **`01_Introduction.md`**: Brief welcome and purpose for users.
|
||||
* **`02_Features.md`**: Detailed list of user-facing features.
|
||||
* **`03_Installation.md`**: Requirements and step-by-step installation instructions.
|
||||
* **`04_Configuration_and_Presets.md`**: Explains user-level configuration options (`config.py` settings relevant to users) and how to select and understand presets.
|
||||
* **`05_Usage_GUI.md`**: Guide on using the Graphical User Interface, including descriptions of panels, controls, and workflow.
|
||||
* **`06_Usage_CLI.md`**: Guide on using the Command-Line Interface, including arguments and examples.
|
||||
* **`07_Usage_Monitor.md`**: Guide on setting up and using the Directory Monitor for automated processing.
|
||||
* **`08_Usage_Blender.md`**: Explains the user-facing aspects of the Blender integration.
|
||||
* **`09_Output_Structure.md`**: Describes the structure and contents of the generated asset library.
|
||||
* **`10_Docker.md`**: Instructions for building and running the tool using Docker.
|
||||
|
||||
### `Documentation/02_Developer_Guide/`
|
||||
|
||||
* **`01_Architecture.md`**: High-level technical architecture, core components, and their relationships.
|
||||
* **`02_Codebase_Structure.md`**: Detailed breakdown of key files and directories within the project.
|
||||
* **`03_Key_Components.md`**: In-depth explanation of major classes and modules (`AssetProcessor`, `Configuration`, GUI Handlers, etc.).
|
||||
* **`04_Configuration_System_and_Presets.md`**: Technical details of the configuration loading and merging process, the structure of preset JSON files, and guidance on creating/modifying presets for developers.
|
||||
* **`05_Processing_Pipeline.md`**: Step-by-step technical breakdown of the asset processing logic within the `AssetProcessor` class.
|
||||
* **`06_GUI_Internals.md`**: Technical details of the GUI implementation, including threading, signals/slots, and background task management.
|
||||
* **`07_Monitor_Internals.md`**: Technical details of the Directory Monitor implementation using `watchdog`.
|
||||
* **`08_Blender_Integration_Internals.md`**: Technical details of how the Blender scripts are executed and interact with the processed assets.
|
||||
* **`09_Development_Workflow.md`**: Guidance for developers on contributing, setting up a development environment, and modifying specific parts of the codebase.
|
||||
* **`10_Coding_Conventions.md`**: Overview of the project's coding standards, object-oriented approach, type hinting, logging, and error handling.
|
||||
* **`11_Debugging_Notes.md`**: Advanced internal details, state management, error propagation, concurrency models, resource management, and known limitations/edge cases.
|
||||
|
||||
This plan provides a solid foundation for organizing the existing documentation and serves as a roadmap for creating the new markdown files.
|
||||
Binary file not shown.
@ -1,356 +0,0 @@
|
||||
# Asset Processor Tool vX.Y
|
||||
|
||||
## Overview
|
||||
|
||||
This tool processes 3D asset source files (texture sets, models, etc., provided as ZIP archives or folders) into a standardized library format. It uses configurable presets to interpret different asset sources and automates tasks like file classification, image resizing, channel merging, and metadata generation.
|
||||
|
||||
The tool offers both a Graphical User Interface (GUI) for interactive use and a Command-Line Interface (CLI) for batch processing and scripting.
|
||||
|
||||
This tool is currently work in progress, rewritting features from an original proof of concept, original script can be found at `Deprecated-POC/` for reference
|
||||
|
||||
## Features
|
||||
|
||||
* **Preset-Driven:** Uses JSON presets (`presets/`) to define rules for different asset suppliers (e.g., `Poliigon.json`).
|
||||
* **Dual Interface:** Provides both a user-friendly GUI and a powerful CLI.
|
||||
* **Parallel Processing:** Utilizes multiple CPU cores for faster processing of multiple assets (configurable via `--workers` in CLI or GUI control).
|
||||
* **Multi-Asset Input Handling:** Correctly identifies and processes multiple distinct assets contained within a single input ZIP or folder, creating separate outputs for each.
|
||||
* **File Classification:** Automatically identifies map types (Color, Normal, Roughness, etc.), models, explicitly marked extra files, and unrecognised files based on preset rules.
|
||||
* **Variant Handling:** Map types listed in `RESPECT_VARIANT_MAP_TYPES` (in `config.py`, e.g., `"COL"`) will *always* receive a numeric suffix (`-1`, `-2`, etc.). The numbering priority is determined primarily by the order of keywords listed in the preset's `map_type_mapping`. Alphabetical sorting of filenames is used only as a tie-breaker for files matching the exact same keyword pattern. Other map types will *never* receive a suffix.
|
||||
* **16-bit Prioritization:** Correctly identifies 16-bit variants defined in preset `bit_depth_variants` (e.g., `*_NRM16.tif`), prioritizes them, and ignores the corresponding 8-bit version (marked as `Ignored` in GUI).
|
||||
* **Map Processing:**
|
||||
* Resizes texture maps to configured power of two resolutions (e.g., 4K, 2K, 1K), avoiding upscaling.
|
||||
* Handles Glossiness map inversion to Roughness.
|
||||
* Applies bit-depth rules (`respect` source or `force_8bit`).
|
||||
* Saves maps in appropriate formats. Map types listed in `FORCE_LOSSLESS_MAP_TYPES` (in `config.py`, e.g., `"NRM"`, `"DISP"`) are *always* saved in a lossless format (PNG for 8-bit, configured 16-bit format like EXR/PNG for 16-bit), overriding other rules. For other map types, if the output is 8-bit and the resolution meets or exceeds `RESOLUTION_THRESHOLD_FOR_JPG` (in `config.py`), the output is forced to JPG. Otherwise, the format is based on input type and target bit depth: JPG inputs yield JPG outputs (8-bit); TIF inputs yield PNG/EXR (based on target bit depth and config); other inputs use configured formats (PNG/EXR). Merged maps follow similar logic, checking `FORCE_LOSSLESS_MAP_TYPES` first, then the threshold for 8-bit targets, then using the highest format from inputs (EXR > TIF > PNG > JPG hierarchy, with TIF adjusted to PNG/EXR based on target bit depth).
|
||||
* Calculates basic image statistics (Min/Max/Mean) for a reference resolution.
|
||||
* Calculates and stores the relative aspect ratio change string in metadata.
|
||||
* **Channel Merging:** Combines channels from different maps into packed textures (e.g., NRMRGH) based on preset rules.
|
||||
* **Metadata Generation:** Creates a `metadata.json` file for each asset containing details about maps, category, archetype, aspect ratio change, processing settings, etc. **Aspect Ratio Metadata:** Calculates the relative aspect ratio change during resizing and stores it in the `metadata.json` file (`aspect_ratio_change_string`). The format indicates if the aspect is unchanged (`EVEN`), scaled horizontally (`X150`, `X110`, etc.), scaled vertically (`Y150`, `Y125`, etc.)
|
||||
* **Output Organization:** Creates a clean, structured output directory (`<output_base>/<supplier>/<asset_name>/`).
|
||||
* **Skip/Overwrite:** Can skip processing if the output already exists or force reprocessing with the `--overwrite` flag (CLI) or checkbox (GUI).
|
||||
* **Blender Integration:** Optionally runs Blender scripts (`create_nodegroups.py`, `create_materials.py`) after asset processing to automate node group and material creation in specified `.blend` files. Available via both CLI and GUI.
|
||||
* **GUI Features:**
|
||||
* Drag-and-drop input for assets (ZIPs/folders).
|
||||
* Integrated preset editor panel for managing `.json` presets.
|
||||
* Configurable output directory field with a browse button (defaults to path in `config.py`).
|
||||
* Enhanced live preview table showing predicted file status (Mapped, Model, Extra, Unrecognised, Ignored, Error) based on the selected processing preset.
|
||||
* Toggleable preview mode (via View menu) to switch between detailed file preview and a simple list of input assets.
|
||||
* Toggleable log console panel (via View menu) displaying application log messages within the GUI.
|
||||
* Progress bar, cancellation button, and clear queue button.
|
||||
* **Blender Post-Processing Controls:** Checkbox to enable/disable Blender script execution and input fields with browse buttons to specify the target `.blend` files for node group and material creation (defaults configurable in `config.py`).
|
||||
* **Responsive GUI:** Utilizes background threads (`QThread`) for processing (`ProcessPoolExecutor`) and file preview generation (`ThreadPoolExecutor`), ensuring the user interface remains responsive during intensive operations.
|
||||
* **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.
|
||||
* **Docker Support:** Includes a `Dockerfile` for containerized execution.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
Asset_processor_tool/
|
||||
│
|
||||
├── main.py # CLI Entry Point & processing orchestrator
|
||||
├── monitor.py # Directory monitoring script for automated processing
|
||||
├── asset_processor.py # Core class handling single asset processing pipeline
|
||||
├── configuration.py # Class for loading and accessing configuration
|
||||
├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)
|
||||
│
|
||||
├── blenderscripts/ # Scripts for integration with Blender
|
||||
│ └── create_nodegroups.py # Script to create node groups from processed assets
|
||||
│ └── create_materials.py # Script to create materials linking to node groups
|
||||
│
|
||||
├── gui/ # Contains files related to the Graphical User Interface
|
||||
│ ├── main_window.py # Main GUI application window and layout
|
||||
│ ├── processing_handler.py # Handles background processing logic for the GUI
|
||||
│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI
|
||||
│
|
||||
├── Presets/ # Preset definition files
|
||||
│ ├── _template.json # Template for creating new presets
|
||||
│ └── Poliigon.json # Example preset for Poliigon assets
|
||||
│
|
||||
├── Testfiles/ # Directory containing example input assets for testing
|
||||
│
|
||||
├── Tickets/ # Directory for issue and feature tracking (Markdown files)
|
||||
│ ├── _template.md # Template for creating new tickets
|
||||
│ └── Ticket-README.md # Explanation of the ticketing system
|
||||
│
|
||||
├── requirements.txt # Python package dependencies for standard execution
|
||||
├── requirements-docker.txt # Dependencies specifically for the Docker environment
|
||||
├── Dockerfile # Instructions for building the Docker container image
|
||||
└── readme.md # This documentation file
|
||||
```
|
||||
|
||||
* **Core Logic:** `main.py`, `monitor.py`, `asset_processor.py`, `configuration.py`, `config.py`
|
||||
* **Blender Integration:** `blenderscripts/` directory
|
||||
* **GUI:** `gui/` directory
|
||||
* **Configuration:** `config.py`, `Presets/` directory
|
||||
* **Dependencies:** `requirements.txt`, `requirements-docker.txt`
|
||||
* **Containerization:** `Dockerfile`
|
||||
* **Documentation/Planning:** `readme.md`, `Project Notes/` directory
|
||||
* **Issue/Feature Tracking:** `Tickets/` directory (see `Tickets/README.md`)
|
||||
* **Testing:** `Testfiles/` directory
|
||||
|
||||
## Architecture
|
||||
|
||||
This section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.
|
||||
|
||||
### Core Components
|
||||
|
||||
The tool is primarily built around several key Python modules:
|
||||
|
||||
* `config.py`: Defines core, global settings (output paths, resolutions, default behaviors, format rules, Blender executable path, default Blender file paths, etc.) that are generally not supplier-specific.
|
||||
* `Presets/*.json`: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).
|
||||
* `configuration.py` **(**`Configuration` **class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.
|
||||
* `asset_processor.py` **(**`AssetProcessor` **class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.
|
||||
* `main.py`: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, calls `AssetProcessor` for each input asset via a wrapper function, and optionally triggers Blender script execution after processing.
|
||||
* `gui/`: Contains modules related to the Graphical User Interface (GUI), built using PySide6.
|
||||
* `monitor.py`: Implements the directory monitoring functionality for automated processing.
|
||||
|
||||
### Parallel Processing (CLI & GUI)
|
||||
|
||||
To accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.
|
||||
|
||||
* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.
|
||||
* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.
|
||||
* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.
|
||||
* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).
|
||||
|
||||
### Asset Processing Pipeline (`AssetProcessor` class)
|
||||
|
||||
The `AssetProcessor` class executes a sequence of steps for each asset:
|
||||
|
||||
1. `_setup_workspace()`: Creates a temporary directory for processing.
|
||||
2. `_extract_input()`: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.
|
||||
3. `_inventory_and_classify_files()`: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:
|
||||
* Identifying files explicitly marked for the `Extra/` folder.
|
||||
* Identifying model files.
|
||||
* Matching potential texture maps against keyword patterns.
|
||||
* Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.
|
||||
* Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.
|
||||
* Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).
|
||||
4. `_determine_base_metadata()`: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).
|
||||
5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.
|
||||
6. `_process_maps()`: Iterates through classified texture maps. For each map:
|
||||
* Loads the image data (handling potential Gloss->Roughness inversion).
|
||||
* Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.
|
||||
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).
|
||||
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:
|
||||
* The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).
|
||||
* The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).
|
||||
* The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).
|
||||
* Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).
|
||||
* The `FORCE_LOSSLESS_MAP_TYPES` list in `config.py` (overrides all other logic for specified map types, ensuring PNG/EXR output).
|
||||
* Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).
|
||||
* Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`) and determines the aspect ratio change string (e.g., "EVEN", "X150", "Y075") stored in the metadata.
|
||||
7. `_merge_maps()`: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps` (checking `FORCE_LOSSLESS_MAP_TYPES` first, then threshold, then input hierarchy), considering the formats of the input maps involved.
|
||||
8. `_generate_metadata_file()`: Collects all gathered information (asset name, maps present, resolutions, stats, aspect ratio change, etc.) and writes it to the `metadata.json` file.
|
||||
9. `_organize_output_files()`: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).
|
||||
10. `_cleanup_workspace()`: Removes the temporary workspace directory.
|
||||
|
||||
### GUI Architecture (`gui/`)
|
||||
|
||||
The GUI provides an interactive way to use the tool and manage presets.
|
||||
|
||||
* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.
|
||||
* **Main Window (**`main_window.py`**)**: Defines the main application window, which includes:
|
||||
* An integrated preset editor panel (using `QSplitter`).
|
||||
* A processing panel with drag-and-drop support, output directory selection, a file preview table, and processing controls.
|
||||
* **Blender Post-Processing Controls:** A group box containing a checkbox to enable/disable Blender script execution and input fields with browse buttons for specifying the target `.blend` files for node group and material creation.
|
||||
* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:
|
||||
* `ProcessingHandler` **(**`processing_handler.py`**)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) and the optional Blender script execution in a background thread. Receives the target output directory and Blender integration settings from the main window.
|
||||
* `PredictionHandler` **(**`prediction_handler.py`**)**: Manages the generation of file previews in a background thread using a `ThreadPoolExecutor` to parallelize prediction across multiple assets. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs extraction and classification.
|
||||
* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors. A custom `QtLogHandler` redirects Python log messages to the UI console via signals.
|
||||
* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending. Includes an optional, toggleable log console panel at the top.
|
||||
|
||||
### Monitor Architecture (`monitor.py`)
|
||||
|
||||
The `monitor.py` script enables automated processing of assets dropped into a designated input directory.
|
||||
|
||||
* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.
|
||||
* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.
|
||||
* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.
|
||||
* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.
|
||||
* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.
|
||||
* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.
|
||||
|
||||
### Error Handling
|
||||
|
||||
* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.
|
||||
* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.
|
||||
* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.
|
||||
|
||||
## Requirements
|
||||
|
||||
* Python 3.8+
|
||||
* Required Python Packages (see `requirements.txt`):
|
||||
* `opencv-python` (for image processing)
|
||||
* `numpy` (for numerical operations)
|
||||
* `PySide6` (only needed for the GUI)
|
||||
* Optional Python Packages:
|
||||
* `OpenEXR` (provides more robust EXR file handling, recommended if processing EXR sources)
|
||||
* **Blender:** A working installation of Blender is required for the optional Blender integration features. The path to the executable should be configured in `config.py` or available in the system's PATH.
|
||||
|
||||
Install dependencies using pip:
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
(For GUI, ensure PySide6 is included or install separately: `pip install PySide6`)
|
||||
|
||||
## Configuration
|
||||
|
||||
The tool's behavior is controlled by two main configuration components:
|
||||
|
||||
1. `config.py`**:** Defines core, global settings:
|
||||
* `OUTPUT_BASE_DIR`: Default root directory for processed assets.
|
||||
* `DEFAULT_ASSET_CATEGORY`: Fallback category ("Texture", "Asset", "Decal").
|
||||
* `IMAGE_RESOLUTIONS`: Dictionary mapping resolution keys (e.g., "4K") to pixel dimensions.
|
||||
* `RESPECT_VARIANT_MAP_TYPES`: List of map type strings (e.g., `["COL"]`) that should always receive a numeric suffix (`-1`, `-2`, etc.) based on preset order, even if only one variant exists.
|
||||
* `TARGET_FILENAME_PATTERN`: Format string for output filenames.
|
||||
* `MAP_MERGE_RULES`: List defining how to merge channels (e.g., creating NRMRGH).
|
||||
* `ARCHETYPE_RULES`: Rules for determining asset usage archetype (e.g., Wood, Metal).
|
||||
* `RESOLUTION_THRESHOLD_FOR_JPG`: Dimension threshold (pixels) above which 8-bit maps are forced to JPG format, overriding other format logic.
|
||||
* `FORCE_LOSSLESS_MAP_TYPES`: List of map type strings (e.g., `["NRM", "DISP"]`) that should *always* be saved losslessly (PNG/EXR), overriding the JPG threshold and other format logic.
|
||||
* `BLENDER_EXECUTABLE_PATH`: Path to the Blender executable (required for Blender integration).
|
||||
* `DEFAULT_NODEGROUP_BLEND_PATH`: Default path to the .blend file for node group creation (used by GUI if not specified).
|
||||
* `DEFAULT_MATERIALS_BLEND_PATH`: Default path to the .blend file for material creation (used by GUI if not specified).
|
||||
* ... and other processing parameters (JPEG quality, PNG compression, 16-bit/8-bit output formats, etc.).
|
||||
2. `presets/*.json`**:** Define supplier-specific rules. Each JSON file represents a preset (e.g., `Poliigon.json`). Key sections include:
|
||||
* `supplier_name`: Name of the asset source.
|
||||
* `map_type_mapping`: A list of dictionaries defining rules to map source filename keywords/patterns to standard map types. Each dictionary should have `"target_type"` (e.g., `"COL"`, `"NRM"`) and `"keywords"` (a list of source filename patterns like `["_col*", "_color"]`). For map types listed in `config.py`'s `RESPECT_VARIANT_MAP_TYPES`, the numbering priority (`-1`, `-2`, etc.) is determined primarily by the order of the keywords within the `"keywords"` list for the matching rule. Alphabetical sorting of filenames is used only as a secondary tie-breaker for files matching the exact same keyword pattern. Other map types do not receive suffixes.
|
||||
* `bit_depth_variants`: Dictionary mapping standard map types (e.g., `"NRM"`) to fnmatch patterns used to identify their high bit-depth source files (e.g., `"*_NRM16*.tif"`). These take priority over standard keyword matches, and the corresponding 8-bit version will be ignored.
|
||||
* `bit_depth_rules`: Specifies whether to `respect` source bit depth or `force_8bit` for specific map types (defined in `config.py`).
|
||||
* `model_patterns`: Regex patterns to identify model files (e.g., `*.fbx`, `*.obj`).
|
||||
* `move_to_extra_patterns`: Regex patterns for files to move directly to the `Extra/` output folder.
|
||||
* `source_naming_convention`: Defines separator and indices for extracting base name/archetype from source filenames.
|
||||
* `asset_category_rules`: Keywords/patterns to identify specific asset categories (e.g., "Decal").
|
||||
|
||||
Use `presets/_template.json` as a starting point for creating new presets.
|
||||
|
||||
## Usage
|
||||
|
||||
### 1. Graphical User Interface (GUI)
|
||||
|
||||
* **Run:**
|
||||
|
||||
```bash
|
||||
python -m gui.main_window
|
||||
```
|
||||
*(Note: Run this command from the project root directory)*
|
||||
* **Interface:**
|
||||
* **Menu Bar:** Contains a "View" menu to toggle visibility of the Log Console and enable/disable the detailed file preview.
|
||||
* **Preset Editor Panel (Left):**
|
||||
* Optional **Log Console:** A text area at the top displaying application log messages (toggle via View menu).
|
||||
* **Preset List:** Allows creating, deleting, loading, editing, and saving presets. Select a preset here to load it into the editor tabs below.
|
||||
* **Preset Editor Tabs:** Edit preset details ("General & Naming", "Mapping & Rules").
|
||||
* **Processing Panel (Right):**
|
||||
* **Preset Selector:** Select the preset to use for *processing* the current queue.
|
||||
* **Output Directory:** Displays the target output directory. Defaults to the path in `config.py`. Use the "Browse..." button to select a different directory.
|
||||
* **Drag and Drop Area:** Drag asset ZIP files or folders here to add them to the queue.
|
||||
* **Preview Table:** Displays information about the assets in the queue. Behavior depends on the "Disable Detailed Preview" option in the View menu:
|
||||
* **Detailed Preview (Default):** Shows all files found within the dropped assets, their predicted classification status (Mapped, Model, Extra, Unrecognised, Ignored, Error), predicted output name (if applicable), and other details based on the selected *processing* preset. Rows are color-coded by status.
|
||||
* **Simple View (Preview Disabled):** Shows only the list of top-level input asset paths (ZIPs/folders) added to the queue.
|
||||
* **Progress Bar:** Shows the overall processing progress.
|
||||
* **Blender Post-Processing:** A group box containing a checkbox to enable/disable the optional Blender script execution. When enabled, input fields and browse buttons appear to specify the `.blend` files for node group and material creation. These fields default to the paths configured in `config.py`.
|
||||
* **Options & Controls (Bottom):**
|
||||
* `Overwrite Existing`: Checkbox to force reprocessing if output already exists.
|
||||
* `Workers`: Spinbox to set the number of assets to process concurrently.
|
||||
* `Clear Queue`: Button to remove all assets from the queue and clear the preview.
|
||||
* `Start Processing`: Button to begin processing all assets in the queue.
|
||||
* `Cancel`: Button to attempt stopping ongoing processing.
|
||||
* **Status Bar:** Displays messages about the current state, errors, or completion.
|
||||
|
||||
### 2. Command-Line Interface (CLI)
|
||||
|
||||
* **Run:**
|
||||
|
||||
```bash
|
||||
python main.py [OPTIONS] INPUT_PATH [INPUT_PATH ...]
|
||||
```
|
||||
* **Arguments:**
|
||||
* `INPUT_PATH`: One or more paths to input ZIP files or folders.
|
||||
* `-p PRESET`, `--preset PRESET`: (Required) Name of the preset to use (e.g., `Poliigon`).
|
||||
* `-o OUTPUT_DIR`, `--output-dir OUTPUT_DIR`: Override the `OUTPUT_BASE_DIR` set in `config.py`.
|
||||
* `-w WORKERS`, `--workers WORKERS`: Number of parallel processes (default: auto-detected based on CPU cores).
|
||||
* `--overwrite`: Force reprocessing and overwrite existing output.
|
||||
* `-v`, `--verbose`: Enable detailed DEBUG level logging.
|
||||
* `--nodegroup-blend NODEGROUP_BLEND`: Path to the .blend file for creating/updating node groups. Overrides `config.py` default. If provided, triggers node group script execution after processing.
|
||||
* `--materials-blend MATERIALS_BLEND`: Path to the .blend file for creating/updating materials. Overrides `config.py` default. If provided, triggers material script execution after processing.
|
||||
* **Example:**
|
||||
|
||||
```bash
|
||||
python main.py "C:/Downloads/WoodFine001.zip" -p Poliigon -o "G:/Assets/Processed" --workers 4 --overwrite --nodegroup-blend "G:/Blender/Libraries/NodeGroups.blend" --materials-blend "G:/Blender/Libraries/Materials.blend"
|
||||
```
|
||||
|
||||
### 3. Directory Monitor (Automated Processing)
|
||||
|
||||
* **Run:**
|
||||
|
||||
```bash
|
||||
python monitor.py
|
||||
```
|
||||
* **Functionality:** This script continuously monitors a specified input directory for new `.zip` files. When a file matching the expected format `[preset]_filename.zip` appears, it automatically triggers the processing pipeline using the extracted preset name. **Note:** The directory monitor currently does *not* support the optional Blender script execution. This feature is only available via the CLI and GUI.
|
||||
* **Configuration (Environment Variables):**
|
||||
* `INPUT_DIR`: Directory to monitor for new ZIP files (default: `/data/input`).
|
||||
* `OUTPUT_DIR`: Base directory for processed asset output (default: `/data/output`).
|
||||
* `PROCESSED_DIR`: Directory where successfully processed/skipped source ZIPs are moved (default: `/data/processed`).
|
||||
* `ERROR_DIR`: Directory where source ZIPs that failed processing are moved (default: `/data/error`).
|
||||
* `LOG_LEVEL`: Logging verbosity (e.g., `INFO`, `DEBUG`) (default: `INFO`).
|
||||
* `POLL_INTERVAL`: How often to check the input directory (seconds) (default: `5`).
|
||||
* `PROCESS_DELAY`: Delay after detecting a file before processing starts (seconds) (default: `2`).
|
||||
* `NUM_WORKERS`: Number of parallel workers for processing (default: auto-detected).
|
||||
* **Output:**
|
||||
* Logs processing activity to the console.
|
||||
* Processed assets are created in the `OUTPUT_DIR` following the standard structure.
|
||||
* The original input `.zip` file is moved to `PROCESSED_DIR` on success/skip or `ERROR_DIR` on failure.
|
||||
|
||||
### 4. Blender Node Group Creation Script (`blenderscripts/create_nodegroups.py`)
|
||||
* **Purpose:** This script, designed to be run *within* Blender (either manually or triggered by `main.py`/GUI), scans processed assets and creates/updates PBR node groups in the active `.blend` file.
|
||||
* **Execution:** Typically run via the Asset Processor tool's CLI or GUI after asset processing. Can also be run manually in Blender's Text Editor.
|
||||
* **Prerequisites (for manual run):**
|
||||
* A library of assets processed by this tool, located at a known path.
|
||||
* A Blender file containing two template node groups named exactly `Template_PBRSET` and `Template_PBRTYPE`.
|
||||
* **Configuration (Inside the script for manual run):**
|
||||
* `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located. This is overridden by the tool when run via CLI/GUI.
|
||||
* **Functionality:** Reads metadata, creates/updates node groups, loads textures, sets up nodes, applies metadata-driven settings (aspect ratio, stats, highest resolution), and sets asset previews. Includes an explicit save command at the end.
|
||||
|
||||
### 5. Blender Material Creation Script (`blenderscripts/create_materials.py`)
|
||||
* **Purpose:** This script, designed to be run *within* Blender (either manually or triggered by `main.py`/GUI), scans processed assets and creates/updates materials in the active `.blend` file that link to the PBRSET node groups created by `create_nodegroups.py`.
|
||||
* **Execution:** Typically run via the Asset Processor tool's CLI or GUI after asset processing. Can also be run manually in Blender's Text Editor.
|
||||
* **Prerequisites (for manual run):**
|
||||
* A library of assets processed by this tool, located at a known path.
|
||||
* A `.blend` file containing the PBRSET node groups created by `create_nodegroups.py`.
|
||||
* A template material in the *current* Blender file named `Template_PBRMaterial` that uses nodes and contains a Group node labeled `PLACEHOLDER_NODE_LABEL`.
|
||||
* **Configuration (Inside the script for manual run):**
|
||||
* `PROCESSED_ASSET_LIBRARY_ROOT`: **Must be updated** within the script to point to the base output directory where the processed supplier folders (e.g., `Poliigon/`) are located. This is overridden by the tool when run via CLI/GUI.
|
||||
* `NODEGROUP_BLEND_FILE_PATH`: **Must be updated** within the script to point to the `.blend` file containing the PBRSET node groups. This is overridden by the tool when run via CLI/GUI.
|
||||
* `TEMPLATE_MATERIAL_NAME`, `PLACEHOLDER_NODE_LABEL`, `MATERIAL_NAME_PREFIX`, `PBRSET_GROUP_PREFIX`, etc., can be adjusted if needed.
|
||||
* **Functionality:** Reads metadata, creates/updates materials by copying the template, links the corresponding PBRSET node group from the specified `.blend` file, marks materials as assets, copies tags, sets custom previews, and sets viewport properties based on metadata. Includes an explicit save command at the end.
|
||||
|
||||
## Processing Pipeline (Simplified)
|
||||
|
||||
1. **Extraction:** Input ZIP/folder contents are extracted/copied to a temporary workspace.
|
||||
2. **Classification:** Files are scanned and classified (map, model, extra, ignored) using preset rules.
|
||||
3. **Metadata Determination:** Asset name, category, and archetype are determined.
|
||||
4. **Skip Check:** If output exists and overwrite is off, processing stops here.
|
||||
5. **Map Processing:** Identified maps are loaded, resized, converted (bit depth, format), and saved. Gloss maps are inverted if needed. Stats are calculated.
|
||||
6. **Merging:** Channels are merged according to preset rules and saved.
|
||||
7. **Metadata Generation:** `metadata.json` is created with all collected information.
|
||||
8. **Output Organization:** Processed files are moved to the final structured output directory.
|
||||
9. **Cleanup:** The temporary workspace is removed.
|
||||
10. **Optional Blender Script Execution:** If configured via CLI or GUI, Blender is launched in the background to run `create_nodegroups.py` and `create_materials.py` on specified `.blend` files, using the processed asset output directory as input.
|
||||
|
||||
## Output Structure
|
||||
|
||||
Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`
|
||||
|
||||
Each asset directory typically contains:
|
||||
|
||||
* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`).
|
||||
* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`).
|
||||
* Model files (if present in source).
|
||||
* `metadata.json`: Detailed information about the asset and processing.
|
||||
* `Extra/` (subdirectory): Contains source files that were not classified as standard maps or models. This includes files explicitly matched by `move_to_extra_patterns` in the preset (e.g., previews, documentation) as well as any other unrecognised files.
|
||||
|
||||
## Docker
|
||||
|
||||
A `Dockerfile` and `requirements-docker.txt` are provided for building a container image to run the processor in an isolated environment. Build and run using standard Docker commands.
|
||||
@ -12,9 +12,9 @@ This documentation strictly excludes details on environment setup, dependency in
|
||||
|
||||
## Architecture and Codebase Summary
|
||||
|
||||
For developers interested in contributing, the tool's architecture is designed around a **Core Processing Engine** (`asset_processor.py`) that handles the pipeline for single assets, supported by a **Configuration System** (`configuration.py` and `config.py` with `Presets/*.json`). Multiple interfaces are provided: a **Graphical User Interface** (`gui/`), a **Command-Line Interface** (`main.py`), and a **Directory Monitor** (`monitor.py`). Optional **Blender Integration** (`blenderscripts/`) is also included.
|
||||
For developers interested in contributing, the tool's architecture centers on a **Core Processing Engine** (`processing_engine.py`) which initializes and runs a **Pipeline Orchestrator** (`processing/pipeline/orchestrator.py::PipelineOrchestrator`). This orchestrator executes a defined sequence of **Processing Stages** (located in `processing/pipeline/stages/`) based on a **Hierarchical Rule System** (`rule_structure.py`) and a **Configuration System** (`configuration.py` loading `config/app_settings.json` and `Presets/*.json`). The **Graphical User Interface** (`gui/`) has been significantly refactored: `MainWindow` (`main_window.py`) acts as a coordinator, delegating tasks to specialized widgets (`MainPanelWidget`, `PresetEditorWidget`, `LogConsoleWidget`) and background handlers (`RuleBasedPredictionHandler`, `LLMPredictionHandler`, `LLMInteractionHandler`, `AssetRestructureHandler`). The **Directory Monitor** (`monitor.py`) now processes archives asynchronously using a thread pool and utility functions (`utils/prediction_utils.py`, `utils/workspace_utils.py`). The **Command-Line Interface** entry point (`main.py`) primarily launches the GUI, with core CLI functionality currently non-operational. Optional **Blender Integration** (`blenderscripts/`) remains. A new `utils/` directory houses shared helper functions.
|
||||
|
||||
The codebase is organized into key directories and files reflecting these components. The `gui/` directory contains all GUI-related code, `Presets/` holds configuration presets, and `blenderscripts/` contains scripts for Blender interaction. The core logic resides in files like `asset_processor.py`, `configuration.py`, `config.py`, `main.py`, and `monitor.py`. The processing pipeline involves steps such as file classification, map processing, channel merging, and metadata generation.
|
||||
The codebase reflects this structure. The `gui/` directory contains the refactored UI components, `utils/` holds shared utilities, `processing/pipeline/` contains the orchestrator and individual processing stages, `Presets/` contains JSON presets, and `blenderscripts/` holds Blender scripts. Core logic resides in `processing_engine.py`, `processing/pipeline/orchestrator.py`, `configuration.py`, `rule_structure.py`, `monitor.py`, and `main.py`. The processing pipeline, initiated by `processing_engine.py` and executed by the `PipelineOrchestrator`, relies entirely on the input `SourceRule` and static configuration. Each stage in the pipeline operates on an `AssetProcessingContext` object (`processing/pipeline/asset_context.py`) to perform specific tasks like map processing, channel merging, and metadata generation.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
|
||||
@ -2,16 +2,60 @@
|
||||
|
||||
This document explains how to configure the Asset Processor Tool and use presets.
|
||||
|
||||
## Core Settings (`config.py`)
|
||||
## Application Settings (`config/app_settings.json`)
|
||||
|
||||
The tool's behavior is controlled by core settings defined in `config.py`. While primarily for developers, some settings are important for users to be aware of:
|
||||
The tool's core settings are now stored in `config/app_settings.json`. This JSON file contains the base configuration for the application.
|
||||
|
||||
* `OUTPUT_BASE_DIR`: The default root directory where processed assets will be saved.
|
||||
* `IMAGE_RESOULTIONS`: Defines the target resolutions for processed texture maps (e.g., 4K, 2K).
|
||||
* `BLENDER_EXECUTABLE_PATH`: The path to your Blender installation, required for optional Blender integration.
|
||||
* Other settings control aspects like default asset category, filename patterns, map merge rules, and output formats.
|
||||
The `configuration.py` module is responsible for loading the settings from `app_settings.json` (including loading and saving the JSON content), merging them with the rules from the selected preset file, and providing the base configuration via the `load_base_config()` function. Note that the old `config.py` file has been deleted.
|
||||
|
||||
These settings can often be overridden via the GUI or CLI arguments.
|
||||
The `app_settings.json` file is structured into several key sections, including:
|
||||
* `FILE_TYPE_DEFINITIONS`: Defines known file types (like different texture maps, models, etc.) and their properties. Each definition now includes a `"standard_type"` key for aliasing to a common type (e.g., "COL" for color maps, "NRM" for normal maps), an `"is_grayscale"` boolean property, and a `"bit_depth_rule"` key specifying how to handle bit depth for this file type. The separate `MAP_BIT_DEPTH_RULES` section has been removed. For users creating or editing presets, it's important to note that internal mapping rules (like `Map_type_Mapping.target_type` within a preset's `FileRule`) now directly use the main keys from these `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_RGH"`), not just the `standard_type` aliases.
|
||||
* `ASSET_TYPE_DEFINITIONS`: Defines known asset types (like Surface, Model, Decal) and their properties.
|
||||
* `MAP_MERGE_RULES`: Defines how multiple input maps can be merged into a single output map (e.g., combining Normal and Roughness into one).
|
||||
|
||||
### LLM Predictor Settings
|
||||
|
||||
For users who wish to utilize the experimental LLM Predictor feature, the following settings are available in `config/llm_settings.json`:
|
||||
|
||||
* `llm_endpoint_url`: The URL of the LLM API endpoint. For local LLMs like LM Studio or Ollama, this will typically be `http://localhost:<port>/v1`. Consult your LLM server documentation for the exact endpoint.
|
||||
* `llm_api_key`: The API key required to access the LLM endpoint. Some local LLM servers may not require a key, in which case this can be left empty.
|
||||
* `llm_model_name`: The name of the specific LLM model to use for prediction. This must match a model available at your specified endpoint.
|
||||
* `llm_temperature`: Controls the randomness of the LLM's output. Lower values (e.g., 0.1-0.5) make the output more deterministic and focused, while higher values (e.g., 0.6-1.0) make it more creative and varied. For prediction tasks, lower temperatures are generally recommended.
|
||||
* `llm_request_timeout`: The maximum time (in seconds) to wait for a response from the LLM API. Adjust this based on the performance of your LLM server and the complexity of the requests.
|
||||
|
||||
Note that the `llm_predictor_prompt` and `llm_predictor_examples` settings are also present in `config/llm_settings.json`. These define the instructions and examples provided to the LLM for prediction. While they can be viewed here, they are primarily intended for developer reference and tuning the LLM's behavior, and most users will not need to modify them directly via the file. These settings are editable via the LLM Editor panel in the main GUI when the LLM interpretation mode is selected.
|
||||
|
||||
## Application Preferences (`config/app_settings.json` overrides)
|
||||
|
||||
You can modify user-overridable application settings using the built-in GUI editor. These settings are loaded from `config/app_settings.json` and saved as overrides in `config/user_settings.json`. Access it via the **Edit** -> **Preferences...** menu.
|
||||
|
||||
This editor provides a tabbed interface to view and change various application behaviors. The tabs include:
|
||||
* **General:** Basic settings like output base directory and temporary file prefix.
|
||||
* **Output & Naming:** Settings controlling output directory and filename patterns, and how variants are handled.
|
||||
* **Image Processing:** Settings related to image resolution definitions, compression levels, and format choices.
|
||||
* **Map Merging:** Configuration for how multiple input maps are combined into single output maps.
|
||||
* **Postprocess Scripts:** Paths to default Blender files for post-processing.
|
||||
|
||||
Note that this editor focuses on user-specific overrides of core application settings. **Asset Type Definitions, File Type Definitions, and Supplier Settings are managed in a separate Definitions Editor.**
|
||||
|
||||
Any changes made through the Preferences editor require an application restart to take effect.
|
||||
|
||||
*(Ideally, a screenshot of the Application Preferences editor would be included here.)*
|
||||
|
||||
## Definitions Editor (`config/asset_type_definitions.json`, `config/file_type_definitions.json`, `config/suppliers.json`)
|
||||
|
||||
Core application definitions that are separate from general user preferences are managed in the dedicated Definitions Editor. This includes defining known asset types, file types, and configuring settings specific to different suppliers. Access it via the **Edit** -> **Edit Definitions...** menu.
|
||||
|
||||
The editor is organized into three tabs:
|
||||
* **Asset Type Definitions:** Define the different categories of assets (e.g., Surface, Model, Decal). For each asset type, you can configure its description, a color for UI representation, and example usage strings.
|
||||
* **File Type Definitions:** Define the specific types of files the tool recognizes (e.g., MAP_COL, MAP_NRM, MODEL). For each file type, you can configure its description, a color, example keywords/patterns, a standard type alias, bit depth handling rules, whether it's grayscale, and an optional keybind for quick assignment in the GUI.
|
||||
* **Supplier Settings:** Configure settings that are specific to assets originating from different suppliers. Currently, this includes the "Normal Map Type" (OpenGL or DirectX) used for normal maps from that supplier.
|
||||
|
||||
Each tab presents a list of the defined items on the left (Asset Types, File Types, or Suppliers). Selecting an item in the list displays its configurable details on the right. Buttons are provided to add new definitions or remove existing ones.
|
||||
|
||||
Changes made in the Definitions Editor are saved directly to their respective configuration files (`config/asset_type_definitions.json`, `config/file_type_definitions.json`, and `config/suppliers.json`). Some changes may require an application restart to take full effect in processing logic.
|
||||
|
||||
*(Ideally, screenshots of the Definitions Editor tabs would be included here.)*
|
||||
|
||||
## Preset Files (`presets/*.json`)
|
||||
|
||||
@ -20,8 +64,18 @@ Preset files define supplier-specific rules for interpreting asset source files.
|
||||
* Presets are located in the `presets/` directory.
|
||||
* Each preset is a JSON file named after the supplier (e.g., `Poliigon.json`).
|
||||
* Presets contain rules based on filename patterns and keywords to identify map types, models, and other files.
|
||||
* They also define how variants (like different resolutions or bit depths) are handled and how asset names and categories are determined from the source filename.
|
||||
* They also define how variants (like different resolutions or bit depths) are handled and how asset names and categories are determined from the source filename. When defining `map_type_mapping` rules within a preset, the `target_type` field must now use a valid key from the `FILE_TYPE_DEFINITIONS` in `config/app_settings.json` (e.g., `"MAP_AO"` instead of a custom alias like `"AO"`).
|
||||
|
||||
When processing assets, you must specify which preset to use. The tool then loads the core settings from `config.py` and merges them with the rules from the selected preset to determine how to process the input.
|
||||
When processing assets, you must specify which preset to use. The tool then loads the core settings from `config/app_settings.json` and merges them with the rules from the selected preset to determine how to process the input.
|
||||
|
||||
A template preset file (`presets/_template.json`) is provided as a base for creating new presets.
|
||||
## Global Output Path Configuration
|
||||
|
||||
The structure and naming of the output files generated by the tool are now controlled by two global settings defined exclusively in `config/app_settings.json`:
|
||||
|
||||
* `OUTPUT_DIRECTORY_PATTERN`: Defines the directory structure where processed assets will be saved.
|
||||
* `OUTPUT_FILENAME_PATTERN`: Defines the naming convention for the individual output files within the generated directory.
|
||||
|
||||
**Important:** These settings are global and apply to all processing tasks, regardless of the selected preset. They are **not** part of individual preset files and cannot be modified using the Preset Editor. You can view and edit these patterns in the main application preferences (**Edit** -> **Preferences...**).
|
||||
|
||||
These patterns use special tokens (e.g., `[assetname]`, `[maptype]`) that are replaced with actual values during processing. For a detailed explanation of how these patterns work together, the available tokens, and examples, please refer to the [Output Structure](./09_Output_Structure.md) section of the User Guide.
|
||||
@ -12,24 +12,51 @@ python -m gui.main_window
|
||||
|
||||
## Interface Overview
|
||||
|
||||
* **Menu Bar:** The "View" menu allows you to toggle the visibility of the Log Console and the Detailed File Preview.
|
||||
* **Menu Bar:** The "Edit" menu contains options to configure application settings and definitions:
|
||||
* **Preferences...:** Opens the Application Preferences editor for user-overridable settings (saved to `config/user_settings.json`).
|
||||
* **Edit Definitions...:** Opens the Definitions Editor for managing Asset Type Definitions, File Type Definitions, and Supplier Settings (saved to their respective files).
|
||||
The "View" menu allows you to toggle the visibility of the Log Console and the Detailed File Preview.
|
||||
* **Preset Editor Panel (Left):**
|
||||
* **Optional Log Console:** Displays application logs (toggle via View menu).
|
||||
* **Preset List:** Create, delete, load, edit, and save presets. On startup, the "-- Select a Preset --" item is explicitly selected. You must select a specific preset from this list to load it into the editor below, enable the detailed file preview, and enable the "Start Processing" button.
|
||||
* **Preset Editor Tabs:** Edit the details of the selected preset.
|
||||
* **Processing Panel (Right):**
|
||||
* **Preset Selector:** Choose the preset to use for *processing* the current queue.
|
||||
* **Output Directory:** Set the output path (defaults to `config.py`, use "Browse...")
|
||||
* **Preset Selector:** Choose the preset to use for *processing* the current queue. (Note: LLM interpretation is now initiated via the right-click context menu in the Preview Table).
|
||||
* **Output Directory:** Set the output path (defaults to `config/app_settings.json`, use "Browse...")
|
||||
* **Drag and Drop Area:** Add asset `.zip`, `.rar`, `.7z` files, or folders by dragging and dropping them here.
|
||||
* **Preview Table:** Shows queued assets. Initially, this area displays a message prompting you to select a preset. Once a preset is selected from the Preset List, the detailed file preview will load here. The mode of the preview depends on the "View" menu:
|
||||
* **Detailed Preview (Default):** Lists all files, predicted status (`Mapped`, `Model`, `Extra`, `Unrecognised`, `Ignored`, `Error`), output name, etc., based on the selected *processing* preset. Text colors are applied to cells based on the status of the individual file they represent. Rows use alternating background colors per asset group for visual separation.
|
||||
* **Simple View (Preview Disabled):** Lists only top-level input asset paths.
|
||||
* **Preview Table:** Shows queued assets in a hierarchical view (Source -> Asset -> File). Assets (files, directories, archives) added via drag-and-drop appear immediately in the table. This table is interactive:
|
||||
* **Editable Fields:** The 'Name' field for Assets and the 'Target Asset', 'Supplier', 'Asset Type', and 'Item Type' fields for all items can be edited directly in the table.
|
||||
* Editing an **Asset Name** automatically updates the 'Target Asset' field for all its child files.
|
||||
* The **Item Type** field is a text input with auto-suggestions based on available types.
|
||||
* **Drag-and-Drop Re-parenting:** File rows can be dragged and dropped onto different Asset rows to change their parent asset association.
|
||||
* **Right-Click Context Menu:** Right-clicking on Source, Asset, or File rows brings up a context menu:
|
||||
* **Re-interpret selected source:** This sub-menu allows re-running the prediction process for the selected source item(s) using either a specific preset or the LLM predictor. The available presets and the "LLM" option are listed dynamically. This replaces the previous standalone "Re-interpret Selected with LLM" button.
|
||||
* **Keybinds for Item Management:** When items are selected in the Preview Table, the following keybinds can be used:
|
||||
* `Ctrl + C`: Sets the file type of selected items to Color/Albedo (`MAP_COL`).
|
||||
* `Ctrl + R`: Toggles the file type of selected items between Roughness (`MAP_ROUGH`) and Glossiness (`MAP_GLOSS`).
|
||||
* `Ctrl + N`: Sets the file type of selected items to Normal (`MAP_NRM`).
|
||||
* `Ctrl + M`: Toggles the file type of selected items between Metalness (`MAP_METAL`) and Reflection/Specular (`MAP_REFL`).
|
||||
* `Ctrl + D`: Sets the file type of selected items to Displacement/Height (`MAP_DISP`).
|
||||
* `Ctrl + E`: Sets the file type of selected items to Extra (`EXTRA`).
|
||||
* `Ctrl + X`: Sets the file type of selected items to Ignore (`FILE_IGNORE`).
|
||||
* `F2`: Prompts to set the asset name for all selected items. This name propagates to the `AssetRule` name or the `FileRule` `target_asset_name_override` for the files under the selected assets. If individual files are selected, it will affect their `target_asset_name_override`.
|
||||
* **Prediction Population:** If a valid preset is selected in the Preset Selector (or if re-interpretation is triggered), the table populates with prediction results as they become available. If no preset is selected, added items show empty prediction fields.
|
||||
* **Columns:** The table displays columns: Name, Target Asset, Supplier, Asset Type, Item Type. The "Target Asset" column stretches to fill available space.
|
||||
* **Coloring:** The *text color* of file items is determined by their Item Type (colors defined in `config/app_settings.json`). The *background color* of file items is a 30% darker shade of their parent asset's background, helping to visually group files within an asset. Asset rows themselves may use alternating background colors based on the application theme.
|
||||
* **Progress Bar:** Shows overall processing progress.
|
||||
* **Blender Post-Processing:** Checkbox to enable Blender scripts. If enabled, shows fields and browse buttons for target `.blend` files (defaults from `config.py`).
|
||||
* **Blender Post-Processing:** Checkbox to enable Blender scripts. If enabled, shows fields and browse buttons for target `.blend` files (defaults from `config/app_settings.json`).
|
||||
* **Options & Controls (Bottom):**
|
||||
* `Overwrite Existing`: Checkbox to force reprocessing.
|
||||
* `Workers`: Spinbox for concurrent processes.
|
||||
* `Clear Queue`: Button to clear the queue and preview.
|
||||
* `Start Processing`: Button to start processing the queue. This button is disabled until a valid preset is selected from the Preset List.
|
||||
* `Start Processing`: Button to start processing the queue. This button is enabled as long as there are items listed in the Preview Table. When clicked, any items that do not have a value assigned in the "Target Asset" column will be automatically ignored for that processing run.
|
||||
* `Cancel`: Button to attempt stopping processing.
|
||||
* **Status Bar:** Displays current status, errors, and completion messages.
|
||||
* **Status Bar:** Displays current status, errors, and completion messages. During LLM processing, the status bar will show messages indicating the progress of the LLM requests.
|
||||
|
||||
## GUI Configuration Editor
|
||||
|
||||
Access the GUI Configuration Editor via the **Edit** -> **Preferences...** menu. This dialog allows you to directly edit the `config/app_settings.json` file, which contains the core application settings. The editor uses a tabbed layout (e.g., "General", "Output & Naming") to organize settings.
|
||||
|
||||
Any changes made in the GUI Configuration Editor require you to restart the application for them to take effect.
|
||||
|
||||
*(Ideally, a screenshot of the GUI Configuration Editor would be included here.)*
|
||||
@ -2,20 +2,65 @@
|
||||
|
||||
This document describes the directory structure and contents of the processed assets generated by the Asset Processor Tool.
|
||||
|
||||
Processed assets are saved to: `<output_base_directory>/<supplier_name>/<asset_name>/`
|
||||
Processed assets are saved to a location determined by two global settings, `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`, defined in `config/app_settings.json`. These settings can be overridden by the user via `config/user_settings.json`.
|
||||
|
||||
* `<output_base_directory>`: The base output directory configured in `config.py` or specified via CLI/GUI.
|
||||
* `<supplier_name>`: The name of the asset supplier, determined from the preset used.
|
||||
* `<asset_name>`: The name of the processed asset, determined from the source filename based on preset rules.
|
||||
* `OUTPUT_DIRECTORY_PATTERN`: Defines the directory structure *within* the Base Output Directory.
|
||||
* `OUTPUT_FILENAME_PATTERN`: Defines the naming convention for individual files *within* the directory created by `OUTPUT_DIRECTORY_PATTERN`.
|
||||
|
||||
These patterns use special tokens (explained below) that are replaced with actual values during processing. You can configure these patterns via the main application preferences (**Edit** -> **Preferences...** -> **Output & Naming** tab). They are global settings and are not part of individual presets.
|
||||
|
||||
### Available Tokens
|
||||
|
||||
The following tokens can be used in both `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`. Note that some tokens make more sense in one pattern than the other (e.g., `[maptype]` and `[ext]` are typically used in the filename pattern).
|
||||
|
||||
* `[Assettype]`: The type of asset (e.g., `Texture`, `Model`, `Surface`).
|
||||
* `[supplier]`: The supplier name (from the preset, e.g., `Poliigon`).
|
||||
* `[assetname]`: The main asset name (e.g., `RustyMetalPanel`).
|
||||
* `[resolution]`: Texture resolution (e.g., `1k`, `2k`, `4k`).
|
||||
* `[ext]`: The output file extension (e.g., `png`, `jpg`, `exr`). (Primarily for filename pattern)
|
||||
* `[IncrementingValue]` or `[####]`: A numerical value that increments based on existing directories matching the `OUTPUT_DIRECTORY_PATTERN` in the output base path. The number of `#` characters determines the zero-padding (e.g., `[###]` -> `001`, `002`). If `[IncrementingValue]` is used, it defaults to 4 digits of padding (`0001`, `0002`).
|
||||
* `[Date]`: Current date (`YYYYMMDD`).
|
||||
* `[Time]`: Current time (`HHMMSS`).
|
||||
* `[Sha5]`: The first 5 characters of the SHA-256 hash of the original input source file (e.g., the source zip archive).
|
||||
* `[ApplicationPath]`: Absolute path to the application directory.
|
||||
* `[maptype]`: The standardized map type identifier (e.g., `COL` for Color/Albedo, `NRM` for Normal, `RGH` for Roughness). This is derived from the `standard_type` defined in the application's `FILE_TYPE_DEFINITIONS` (managed in `config/file_type_definitions.json` via the Definitions Editor) and may include a variant suffix if applicable. (Primarily for filename pattern)
|
||||
* `[dimensions]`: Pixel dimensions (e.g., `2048x2048`).
|
||||
* `[bitdepth]`: Output bit depth (e.g., `8bit`, `16bit`).
|
||||
* `[category]`: Asset category determined by preset rules.
|
||||
* `[archetype]`: Asset archetype determined by preset rules.
|
||||
* `[variant]`: Asset variant identifier determined by preset rules.
|
||||
* `[source_filename]`: The original filename of the source file being processed.
|
||||
* `[source_basename]`: The original filename without the extension.
|
||||
* `[source_dirname]`: The directory containing the original source file.
|
||||
|
||||
### Example Output Paths
|
||||
|
||||
The final output path is constructed by combining the Base Output Directory (set in Preferences or via CLI) with the results of the two patterns.
|
||||
|
||||
**Example 1:**
|
||||
|
||||
* Base Output Directory: `/home/user/ProcessedAssets`
|
||||
* `OUTPUT_DIRECTORY_PATTERN`: `[supplier]/[assetname]/[resolution]`
|
||||
* `OUTPUT_FILENAME_PATTERN`: `[assetname]_[maptype]_[resolution].[ext]`
|
||||
* Resulting Path for an Albedo map: `/home/user/ProcessedAssets/Poliigon/WoodFloor001/4k/WoodFloor001_Albedo_4k.png`
|
||||
|
||||
**Example 2:**
|
||||
|
||||
* Base Output Directory: `Output` (relative path)
|
||||
* `OUTPUT_DIRECTORY_PATTERN`: `[Assettype]/[category]/[assetname]`
|
||||
* `OUTPUT_FILENAME_PATTERN`: `[maptype].[ext]`
|
||||
* Resulting Path for a Normal map: `Output/Texture/Wood/WoodFloor001/Normal.exr`
|
||||
|
||||
The `<output_base_directory>` (the root folder where processing output starts) is configured separately via the GUI (**Edit** -> **Preferences...** -> **General** tab -> **Output Base Directory**) or the `--output` CLI argument. The `OUTPUT_DIRECTORY_PATTERN` defines the structure *within* this base directory, and `OUTPUT_FILENAME_PATTERN` defines the filenames within that structure.
|
||||
|
||||
## Contents of Each Asset Directory
|
||||
|
||||
Each asset directory contains the following:
|
||||
|
||||
* Processed texture maps (e.g., `AssetName_Color_4K.png`, `AssetName_NRM_2K.exr`). These are the resized, format-converted, and bit-depth adjusted texture files.
|
||||
* Merged texture maps (e.g., `AssetName_NRMRGH_4K.png`). These are maps created by combining channels from different source maps based on the configured merge rules.
|
||||
* Processed texture maps (e.g., `WoodFloor_Albedo_4k.png`, `MetalPanel_Normal_2k.exr`). The exact filenames depend on the `OUTPUT_FILENAME_PATTERN`. These are the resized, format-converted, and bit-depth adjusted texture files.
|
||||
* Merged texture maps (e.g., `WoodFloor_Combined_4k.png`). The exact filenames depend on the `OUTPUT_FILENAME_PATTERN`. These are maps created by combining channels from different source maps based on the configured merge rules.
|
||||
* Model files (if present in the source asset).
|
||||
* `metadata.json`: A JSON file containing detailed information about the asset and the processing that was performed. This includes details about the maps, resolutions, formats, bit depths, merged map details, calculated image statistics, aspect ratio change information, asset category and archetype, the source preset used, and a list of ignored source files. This file is intended for use by downstream tools or scripts (like the Blender integration scripts).
|
||||
* `Extra/` (subdirectory): Contains source files that were not classified as maps or models but were explicitly marked to be moved to the extra directory based on preset rules (e.g., previews, documentation files).
|
||||
* `metadata.json`: A JSON file containing detailed information about the asset and the processing that was performed. This includes details about the maps (resolutions, formats, bit depths, and for roughness maps, a `derived_from_gloss_filename: true` flag if it was inverted from an original gloss map), merged map details, calculated image statistics, aspect ratio change information, asset category and archetype, the source preset used, and a list of ignored source files. This file is intended for use by downstream tools or scripts (like the Blender integration scripts).
|
||||
* `EXTRA/` (subdirectory): Contains source files not classified as maps or models but marked as "EXTRA" by preset rules (e.g., previews, documentation). These files are placed in an `EXTRA` folder *within* the directory generated by `OUTPUT_DIRECTORY_PATTERN`.
|
||||
* `Unrecognised/` (subdirectory): Contains source files that were not classified as maps, models, or explicitly marked as extra, and were not ignored.
|
||||
* `Ignored/` (subdirectory): Contains source files that were explicitly ignored during processing (e.g., an 8-bit Normal map when a 16-bit variant exists and is prioritized).
|
||||
83
Documentation/01_User_Guide/11_Usage_Autotest.md
Normal file
83
Documentation/01_User_Guide/11_Usage_Autotest.md
Normal file
@ -0,0 +1,83 @@
|
||||
# User Guide: Usage - Automated GUI Testing (`autotest.py`)
|
||||
|
||||
This document explains how to use the `autotest.py` script for automated sanity checks of the Asset Processor Tool's GUI-driven workflow.
|
||||
|
||||
## Overview
|
||||
|
||||
The `autotest.py` script provides a way to run predefined test scenarios headlessly (without displaying the GUI). It simulates the core user actions: loading an asset, selecting a preset, allowing rules to be predicted, processing the asset, and then checks the results against expectations. This is primarily intended as a developer tool for regression testing and ensuring core functionality remains stable.
|
||||
|
||||
## Running the Autotest Script
|
||||
|
||||
From the project root directory, you can run the script using Python:
|
||||
|
||||
```bash
|
||||
python autotest.py [OPTIONS]
|
||||
```
|
||||
|
||||
### Command-Line Options
|
||||
|
||||
The script accepts several command-line arguments to configure the test run. If not provided, they use predefined default values.
|
||||
|
||||
* `--zipfile PATH_TO_ZIP`:
|
||||
* Specifies the path to the input asset `.zip` file to be used for the test.
|
||||
* Default: `TestFiles/BoucleChunky001.zip`
|
||||
* `--preset PRESET_NAME`:
|
||||
* Specifies the name of the preset to be selected and used for rule prediction and processing.
|
||||
* Default: `Dinesen`
|
||||
* `--expectedrules PATH_TO_JSON`:
|
||||
* Specifies the path to a JSON file containing the expected rule structure that should be generated after the preset is applied to the input asset.
|
||||
* Default: `TestFiles/test-BoucleChunky001.json`
|
||||
* `--outputdir PATH_TO_DIR`:
|
||||
* Specifies the directory where the processed assets will be written.
|
||||
* Default: `TestFiles/TestOutputs/DefaultTestOutput`
|
||||
* `--search "SEARCH_TERM"` (optional):
|
||||
* A string to search for within the application logs generated during the test run. If found, matching log lines (with context) will be highlighted.
|
||||
* Default: None
|
||||
* `--additional-lines NUM_LINES` (optional):
|
||||
* When using `--search`, this specifies how many lines of context before and after each matching log line should be displayed.
|
||||
* Default: `0`
|
||||
|
||||
**Example Usage:**
|
||||
|
||||
```bash
|
||||
# Run with default test files and settings
|
||||
python autotest.py
|
||||
|
||||
# Run with specific test files and search for a log message
|
||||
python autotest.py --zipfile TestFiles/MySpecificAsset.zip --preset MyPreset --expectedrules TestFiles/MySpecificAsset_rules.json --outputdir TestFiles/TestOutputs/MySpecificOutput --search "Processing complete for asset"
|
||||
```
|
||||
|
||||
## `TestFiles` Directory
|
||||
|
||||
The autotest script relies on a directory named `TestFiles` located in the project root. This directory should contain:
|
||||
|
||||
* **Test Asset `.zip` files:** The actual asset archives used as input for tests (e.g., `default_test_asset.zip`, `MySpecificAsset.zip`).
|
||||
* **Expected Rules `.json` files:** JSON files defining the expected rule structure for a given asset and preset combination (e.g., `default_test_asset_rules.json`, `MySpecificAsset_rules.json`). The structure of this file is detailed in the main autotest plan (`AUTOTEST_GUI_PLAN.md`).
|
||||
* **`TestOutputs/` subdirectory:** This is the default parent directory where the autotest script will create specific output folders for each test run (e.g., `TestFiles/TestOutputs/DefaultTestOutput/`).
|
||||
|
||||
## Test Workflow
|
||||
|
||||
When executed, `autotest.py` performs the following steps:
|
||||
|
||||
1. **Initialization:** Parses command-line arguments and initializes the main application components headlessly.
|
||||
2. **Load Expected Rules:** Loads the `expected_rules.json` file.
|
||||
3. **Load Asset:** Loads the specified `.zip` file into the application.
|
||||
4. **Select Preset:** Selects the specified preset. This triggers the internal rule prediction process.
|
||||
5. **Await Prediction:** Waits for the rule prediction to complete.
|
||||
6. **Compare Rules:** Retrieves the predicted rules from the application and compares them against the loaded expected rules. If there's a mismatch, the test typically fails at this point.
|
||||
7. **Start Processing:** If the rules match, it initiates the asset processing pipeline, directing output to the specified output directory.
|
||||
8. **Await Processing:** Waits for all backend processing tasks to complete.
|
||||
9. **Check Output:** Verifies the existence of the output directory and lists its contents. Basic checks ensure some output was generated.
|
||||
10. **Analyze Logs:** Retrieves logs from the application. If a search term was provided, it filters and displays relevant log portions. It also checks for Python tracebacks, which usually indicate a failure.
|
||||
11. **Report Result:** Prints a summary of the test outcome (success or failure) and exits with an appropriate status code (0 for success, 1 for failure).
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
* **Console Output:** The script will log its progress and the results of each step to the console.
|
||||
* **Log Analysis:** Pay attention to the log output, especially if a `--search` term was used or if any tracebacks are reported.
|
||||
* **Exit Code:**
|
||||
* `0`: Test completed successfully.
|
||||
* `1`: Test failed at some point (e.g., rule mismatch, processing error, traceback found).
|
||||
* **Output Directory:** Inspect the contents of the specified output directory to manually verify the processed assets if needed.
|
||||
|
||||
This automated test helps ensure the stability of the core processing logic when driven by GUI-equivalent actions.
|
||||
@ -6,41 +6,76 @@ This document provides a high-level overview of the Asset Processor Tool's archi
|
||||
|
||||
The Asset Processor Tool is designed to process 3D asset source files into a standardized library format. Its high-level architecture consists of:
|
||||
|
||||
1. **Core Processing Engine (`AssetProcessor`):** The central component responsible for orchestrating the asset processing pipeline for a single input asset.
|
||||
2. **Configuration System (`Configuration`):** Handles loading core settings and merging them with supplier-specific rules defined in JSON presets.
|
||||
3. **Multiple Interfaces:** Provides different ways to interact with the tool:
|
||||
1. **Core Processing Initiation (`processing_engine.py`):** The `ProcessingEngine` class acts as the entry point for an asset processing task. It initializes and runs a `PipelineOrchestrator`.
|
||||
2. **Pipeline Orchestration (`processing/pipeline/orchestrator.py`):** The `PipelineOrchestrator` manages a sequence of discrete processing stages. It creates an `AssetProcessingContext` for each asset and passes this context through each stage.
|
||||
3. **Processing Stages (`processing/pipeline/stages/`):** Individual modules, each responsible for a specific task in the pipeline (e.g., filtering files, processing maps, merging channels, organizing output). They operate on the `AssetProcessingContext`.
|
||||
4. **Prediction System:** Responsible for analyzing input files and generating the initial `SourceRule` hierarchy with predicted values. This system utilizes a base handler (`gui/base_prediction_handler.py::BasePredictionHandler`) with specific implementations:
|
||||
* **Rule-Based Predictor (`gui/prediction_handler.py::RuleBasedPredictionHandler`):** Uses predefined rules from presets to classify files and determine initial processing parameters.
|
||||
* **LLM Predictor (`gui/llm_prediction_handler.py::LLMPredictionHandler`):** An experimental alternative that uses a Large Language Model (LLM) to interpret file contents and context to predict processing parameters.
|
||||
5. **Configuration System (`Configuration`):** Handles loading core settings (including centralized type definitions and LLM-specific configuration) and merging them with supplier-specific rules defined in JSON presets and the persistent `config/suppliers.json` file.
|
||||
6. **Multiple Interfaces:** Provides different ways to interact with the tool:
|
||||
* Graphical User Interface (GUI)
|
||||
* Command-Line Interface (CLI)
|
||||
* Command-Line Interface (CLI) - *Note: The primary CLI execution logic (`run_cli` in `main.py`) is currently non-functional/commented out post-refactoring.*
|
||||
* Directory Monitor for automated processing.
|
||||
These interfaces exchange data structures containing information about each file and asset set being processed.
|
||||
4. **Optional Integration:** Includes scripts and logic for integrating with external software, specifically Blender, to automate material and node group creation.
|
||||
The GUI acts as the primary source of truth for processing rules, coordinating the generation and management of the `SourceRule` hierarchy before sending it to the `ProcessingEngine`. It accumulates prediction results from multiple input sources before updating the view. The Monitor interface can also generate `SourceRule` objects (using `utils/prediction_utils.py`) to bypass the GUI for automated workflows.
|
||||
7. **Optional Integration:** Includes scripts (`blenderscripts/`) for integrating with Blender. Logic for executing these scripts was intended to be centralized in `utils/blender_utils.py`, but this utility has not yet been implemented.
|
||||
|
||||
## Hierarchical Rule System
|
||||
|
||||
A key addition to the architecture is the **Hierarchical Rule System**, which provides a dynamic layer of configuration that can override the static settings loaded from presets. This system is represented by a hierarchy of rule objects: `SourceRule`, `AssetRule`, and `FileRule`.
|
||||
|
||||
* **SourceRule:** Represents rules applied at the top level, typically corresponding to an entire input source (e.g., a ZIP file or folder).
|
||||
* **AssetRule:** Represents rules applied to a specific asset within a source (a source can contain multiple assets).
|
||||
* **FileRule:** Represents rules applied to individual files within an asset.
|
||||
|
||||
This hierarchy allows for fine-grained control over processing parameters. The GUI's prediction logic generates this hierarchy with initial predicted values for overridable fields based on presets and file analysis. The `ProcessingEngine` (via the `PipelineOrchestrator` and its stages) then operates *solely* on the explicit values provided in this `SourceRule` object and static configuration, without internal prediction or fallback logic.
|
||||
|
||||
## Core Components
|
||||
|
||||
* `config.py`: Defines core, global settings and constants.
|
||||
* `Presets/*.json`: Supplier-specific JSON files defining rules for file interpretation and processing.
|
||||
* `configuration.py` (`Configuration` class): Loads `config.py` settings and merges them with a selected preset, pre-compiling regex patterns for efficiency.
|
||||
* `asset_processor.py` (`AssetProcessor` class): Contains the core logic for processing a *single* asset through the defined pipeline steps. This component works with data structures containing detailed information about the asset and its files.
|
||||
* `main.py`: The entry point for the Command-Line Interface (CLI). It handles argument parsing, logging, parallel processing orchestration, and triggering Blender scripts. It orchestrates the processing of multiple assets by interacting with the `AssetProcessor` for individual assets and manages the overall CLI execution flow.
|
||||
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The GUI interacts with the core processing logic indirectly via dedicated handler classes (`ProcessingHandler`, `PredictionHandler`) running in separate threads. Data structures, such as input paths, configuration details, processing progress updates, and file prediction results, are passed between the GUI, these handlers, and the `AssetProcessor` using thread-safe mechanisms like Qt signals and slots.
|
||||
* `monitor.py`: Implements the directory monitoring feature using `watchdog`.
|
||||
* `config/app_settings.json`: Defines core, global settings, constants, and centralized definitions for allowed asset and file types (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`), including metadata like colors and descriptions. This replaces the old `config.py` file.
|
||||
* `config/suppliers.json`: A persistent JSON file storing known supplier names for GUI auto-completion.
|
||||
* `Presets/*.json`: Supplier-specific JSON files defining rules for file interpretation and initial prediction.
|
||||
* `configuration.py` (`Configuration` class): Loads `config/app_settings.json` settings and merges them with a selected preset, pre-compiling regex patterns for efficiency. This static configuration is used by the processing pipeline.
|
||||
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses used to represent the hierarchical processing rules.
|
||||
* `gui/`: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The `MainWindow` (`main_window.py`) acts as a coordinator, orchestrating interactions between various components. Key GUI components include:
|
||||
* `main_panel_widget.py::MainPanelWidget`: Contains the primary controls for loading sources, selecting presets, viewing/editing rules, and initiating processing.
|
||||
* `preset_editor_widget.py::PresetEditorWidget`: Provides the interface for managing presets.
|
||||
* `log_console_widget.py::LogConsoleWidget`: Displays application logs.
|
||||
* `unified_view_model.py::UnifiedViewModel`: Implements the `QAbstractItemModel` for the hierarchical rule view, holding `SourceRule` data and managing display logic (coloring, etc.). Caches configuration data for performance.
|
||||
* `rule_hierarchy_model.py::RuleHierarchyModel`: A simpler model used internally by the `UnifiedViewModel` to manage the `SourceRule` data structure.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations for inline editing in the rule view.
|
||||
* `asset_restructure_handler.py::AssetRestructureHandler`: Handles complex model updates when a file's target asset is changed via the GUI, ensuring the `SourceRule` hierarchy is correctly modified.
|
||||
* `base_prediction_handler.py::BasePredictionHandler`: Abstract base class for prediction logic.
|
||||
* `prediction_handler.py::RuleBasedPredictionHandler`: Generates the initial `SourceRule` hierarchy based on presets and file analysis. Inherits from `BasePredictionHandler`.
|
||||
* `llm_prediction_handler.py::LLMPredictionHandler`: Experimental predictor using an LLM. Inherits from `BasePredictionHandler`.
|
||||
* `llm_interaction_handler.py::LLMInteractionHandler`: Manages communication with the LLM service for the LLM predictor.
|
||||
* `processing_engine.py` (`ProcessingEngine` class): The entry-point class that initializes and runs the `PipelineOrchestrator` for a given `SourceRule` and `Configuration`.
|
||||
* `processing/pipeline/orchestrator.py` (`PipelineOrchestrator` class): Manages the sequence of processing stages, creating and passing an `AssetProcessingContext` through them.
|
||||
* `processing/pipeline/asset_context.py` (`AssetProcessingContext` class): A dataclass holding all data and state for the processing of a single asset, passed between stages.
|
||||
* `processing/pipeline/stages/`: Directory containing individual processing stage modules, each handling a specific part of the pipeline (e.g., `IndividualMapProcessingStage`, `MapMergingStage`).
|
||||
* `main.py`: The main entry point for the application. Primarily launches the GUI. Contains commented-out/non-functional CLI logic (`run_cli`).
|
||||
* `monitor.py`: Implements the directory monitoring feature using `watchdog`. It now processes archives asynchronously using a `ThreadPoolExecutor`, leveraging `utils.prediction_utils.py` for rule generation and `utils.workspace_utils.py` for workspace management before invoking the `ProcessingEngine`.
|
||||
* `blenderscripts/`: Contains Python scripts designed to be executed *within* Blender for post-processing tasks.
|
||||
* `utils/`: Directory containing utility modules:
|
||||
* `workspace_utils.py`: Contains functions like `prepare_processing_workspace` for handling temporary directories and archive extraction.
|
||||
* `prediction_utils.py`: Contains functions like `generate_source_rule_from_archive` used by the monitor for rule-based prediction.
|
||||
* `blender_utils.py`: (Intended location for Blender script execution logic, currently not implemented).
|
||||
|
||||
## Processing Pipeline (Simplified)
|
||||
## Processing Pipeline (Simplified Overview)
|
||||
|
||||
The core processing engine (`AssetProcessor`) executes a series of steps for each asset:
|
||||
The asset processing pipeline, initiated by `processing_engine.py` and managed by `PipelineOrchestrator`, executes a series of stages for each asset defined in the `SourceRule`. An `AssetProcessingContext` object carries data between stages. The typical sequence is:
|
||||
|
||||
1. Extraction of input to a temporary workspace.
|
||||
2. Classification of files (map, model, extra, ignored, unrecognised) using preset rules.
|
||||
3. Determination of base metadata (asset name, category, archetype).
|
||||
4. Skip check if output exists and overwrite is not forced.
|
||||
5. Processing of maps (resize, format/bit depth conversion, inversion, stats calculation).
|
||||
6. Merging of channels based on rules.
|
||||
7. Generation of `metadata.json` file.
|
||||
8. Organization of processed files into the final output structure.
|
||||
9. Cleanup of the temporary workspace.
|
||||
10. (Optional) Execution of Blender scripts for post-processing.
|
||||
1. **Supplier Determination**: Identify the effective supplier.
|
||||
2. **Asset Skip Logic**: Check if the asset should be skipped.
|
||||
3. **Metadata Initialization**: Set up initial asset metadata.
|
||||
4. **File Rule Filtering**: Determine which files to process.
|
||||
5. **Pre-Map Processing**:
|
||||
* Gloss-to-Roughness Conversion.
|
||||
* Alpha Channel Extraction.
|
||||
* Normal Map Green Channel Inversion.
|
||||
6. **Individual Map Processing**: Handle individual maps (scaling, variants, stats, naming).
|
||||
7. **Map Merging**: Combine channels from different maps.
|
||||
8. **Metadata Finalization & Save**: Generate and save `metadata.json` (temporarily).
|
||||
9. **Output Organization**: Copy all processed files to final output locations.
|
||||
|
||||
This architecture allows for a modular design, separating configuration, core processing logic, and different interfaces. The data structures flowing through this pipeline carry detailed information about individual files and asset sets. Examples include lists of input file paths, configuration objects, dictionaries summarizing processing outcomes, and detailed lists of file predictions. Parallel processing is utilized for efficiency, and background threads keep the GUI responsive.
|
||||
|
||||
**Note on Data Passing:** Major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. These changes are expected to involve new data structures and updated interaction patterns to convey detailed specifications for datasets/asset-sets and processing instructions for individual files. The documentation in this section, particularly regarding data flow, will require significant review and updates once the plan for these changes is finalized.
|
||||
External steps like workspace preparation/cleanup and optional Blender script execution bracket this core pipeline. This architecture allows for a modular design, separating configuration, rule generation/management, and core processing execution.
|
||||
@ -4,61 +4,90 @@ This document outlines the key files and directories within the Asset Processor
|
||||
|
||||
```
|
||||
Asset_processor_tool/
|
||||
├── asset_processor.py # Core class handling single asset processing pipeline
|
||||
├── config.py # Core settings definition (output paths, resolutions, merge rules etc.)
|
||||
├── configuration.py # Class for loading and accessing configuration (merges config.py and presets)
|
||||
├── detailed_documentation_plan.md # (Existing file, potentially outdated)
|
||||
├── configuration.py # Class for loading and accessing configuration (merges app_settings.json and presets)
|
||||
├── Dockerfile # Instructions for building the Docker container image
|
||||
├── documentation_plan.md # Plan for the new documentation structure (this plan)
|
||||
├── documentation.txt # Original developer documentation (to be migrated)
|
||||
├── main.py # CLI Entry Point & processing orchestrator
|
||||
├── monitor.py # Directory monitoring script for automated processing
|
||||
├── readme.md # Original main documentation file (to be migrated)
|
||||
├── readme.md.bak # Backup of readme.md
|
||||
├── main.py # Main application entry point (primarily GUI launcher)
|
||||
├── monitor.py # Directory monitoring script for automated processing (async)
|
||||
├── processing_engine.py # Core class handling single asset processing based on SourceRule
|
||||
├── requirements-docker.txt # Dependencies specifically for the Docker environment
|
||||
├── requirements.txt # Python package dependencies for standard execution
|
||||
├── rule_structure.py # Dataclasses for hierarchical rules (SourceRule, AssetRule, FileRule)
|
||||
├── blenderscripts/ # Scripts for integration with Blender
|
||||
│ ├── create_materials.py # Script to create materials linking to node groups
|
||||
│ └── create_nodegroups.py # Script to create node groups from processed assets
|
||||
├── Deprecated-POC/ # Directory containing original proof of concept scripts
|
||||
│ ├── Blender-MaterialsFromNodegroups.py
|
||||
│ ├── Blender-NodegroupsFromPBRSETS.py
|
||||
│ └── Standalonebatcher-Main.py
|
||||
├── Documentation/ # New directory for organized documentation (this structure)
|
||||
├── config/ # Directory for configuration files
|
||||
│ ├── app_settings.json # Core settings, constants, and type definitions
|
||||
│ └── suppliers.json # Persistent list of known supplier names for GUI auto-completion
|
||||
├── Deprecated/ # Contains old code, documentation, and POC scripts
|
||||
│ ├── ...
|
||||
├── Documentation/ # Directory for organized documentation (this structure)
|
||||
│ ├── 00_Overview.md
|
||||
│ ├── 01_User_Guide/
|
||||
│ └── 02_Developer_Guide/
|
||||
├── gui/ # Contains files related to the Graphical User Interface
|
||||
│ ├── main_window.py # Main GUI application window and layout
|
||||
│ ├── processing_handler.py # Handles background processing logic for the GUI
|
||||
│ ├── prediction_handler.py # Handles background file prediction/preview for the GUI
|
||||
│ ├── preview_table_model.py # Model and proxy for the GUI's preview table
|
||||
│ └── ... # Other GUI components
|
||||
├── Presets/ # Preset definition files
|
||||
├── gui/ # Contains files related to the Graphical User Interface (PySide6)
|
||||
│ ├── asset_restructure_handler.py # Handles model updates for target asset changes
|
||||
│ ├── base_prediction_handler.py # Abstract base class for prediction logic
|
||||
│ ├── config_editor_dialog.py # Dialog for editing configuration files
|
||||
│ ├── delegates.py # Custom delegates for inline editing in rule view
|
||||
│ ├── llm_interaction_handler.py # Manages communication with LLM service
|
||||
│ ├── llm_prediction_handler.py # LLM-based prediction handler
|
||||
│ ├── log_console_widget.py # Widget for displaying logs
|
||||
│ ├── main_panel_widget.py # Main panel containing core GUI controls
|
||||
│ ├── main_window.py # Main GUI application window (coordinator)
|
||||
│ ├── prediction_handler.py # Rule-based prediction handler
|
||||
│ ├── preset_editor_widget.py # Widget for managing presets
|
||||
│ ├── preview_table_model.py # Model for the (deprecated?) preview table
|
||||
│ ├── rule_editor_widget.py # Widget containing the rule hierarchy view and editor
|
||||
│ ├── rule_hierarchy_model.py # Internal model for rule hierarchy data
|
||||
│ └── unified_view_model.py # QAbstractItemModel for the rule hierarchy view
|
||||
├── llm_prototype/ # Files related to the experimental LLM predictor prototype
|
||||
│ ├── ...
|
||||
├── Presets/ # Preset definition files (JSON)
|
||||
│ ├── _template.json # Template for creating new presets
|
||||
│ ├── Poliigon.json # Example preset for Poliigon assets
|
||||
│ └── ... # Other presets
|
||||
├── Project Notes/ # Directory for issue and feature tracking (Markdown files)
|
||||
│ ├── ... # Various planning and note files
|
||||
└── Testfiles/ # Directory containing example input assets for testing
|
||||
└── ... # Example asset ZIPs
|
||||
├── ProjectNotes/ # Directory for developer notes, plans, etc. (Markdown files)
|
||||
│ ├── ...
|
||||
├── PythonCheatsheats/ # Utility Python reference files
|
||||
│ ├── ...
|
||||
├── Testfiles/ # Directory containing example input assets for testing
|
||||
│ ├── ...
|
||||
├── Tickets/ # Directory for issue and feature tracking (Markdown files)
|
||||
│ ├── ...
|
||||
└── utils/ # Utility modules shared across the application
|
||||
├── prediction_utils.py # Utilities for prediction (e.g., used by monitor)
|
||||
└── workspace_utils.py # Utilities for managing processing workspaces
|
||||
```
|
||||
|
||||
**Key Files and Directories:**
|
||||
|
||||
* `asset_processor.py`: Contains the `AssetProcessor` class, the core logic for processing a single asset through the pipeline. Includes methods for classification, map processing, merging, metadata generation, and output organization. Also provides methods for predicting output structure used by the GUI.
|
||||
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config.py` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency.
|
||||
* `config.py`: Stores global default settings, constants, and core rules (e.g., standard map types, default resolutions, merge rules, output format rules, Blender paths).
|
||||
* `main.py`: Entry point for the Command-Line Interface (CLI). Handles argument parsing, logging setup, parallel processing orchestration (using `concurrent.futures.ProcessPoolExecutor`), calls `AssetProcessor` via a wrapper function, and optionally triggers Blender scripts.
|
||||
* `monitor.py`: Implements the automated directory monitoring feature using the `watchdog` library. Contains the `ZipHandler` class to detect new ZIP files and trigger processing via `main.run_processing`.
|
||||
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6.
|
||||
* `main_window.py`: Defines the `MainWindow` class, the main application window structure, UI layout, event handling, and menu setup. Manages GUI-specific logging (`QtLogHandler`).
|
||||
* `processing_handler.py`: Defines the `ProcessingHandler` class (runs on a `QThread`). Manages the execution of the main asset processing pipeline and Blender script execution in the background.
|
||||
* `prediction_handler.py`: Defines the `PredictionHandler` class (runs on a `QThread`). Manages background file analysis/preview generation.
|
||||
* `preview_table_model.py`: Defines `PreviewTableModel` and `PreviewSortFilterProxyModel` for managing and displaying data in the GUI's preview table.
|
||||
* `config/`: Directory containing configuration files.
|
||||
* `app_settings.json`: Stores global default settings, constants, core rules, and centralized definitions for allowed asset and file types (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`) used for validation, GUI elements, and coloring. Replaces the old `config.py`.
|
||||
* `suppliers.json`: A JSON file storing a persistent list of known supplier names, used by the GUI for auto-completion.
|
||||
* `configuration.py`: Defines the `Configuration` class. Responsible for loading core settings from `config/app_settings.json` and merging them with a specified preset JSON file (`Presets/*.json`). Pre-compiles regex patterns from presets for efficiency. An instance of this class is passed to the `ProcessingEngine`.
|
||||
* `rule_structure.py`: Defines the `SourceRule`, `AssetRule`, and `FileRule` dataclasses. These structures represent the hierarchical processing rules and are the primary data contract passed from the rule generation layer (GUI, Monitor) to the processing engine.
|
||||
* `processing_engine.py`: Defines the `ProcessingEngine` class. This is the core component that executes the processing pipeline for a single asset based *solely* on a provided `SourceRule` object and the static `Configuration`. It contains no internal prediction or fallback logic.
|
||||
* `main.py`: Main entry point for the application. Primarily responsible for initializing and launching the GUI (`gui.main_window.MainWindow`). Contains non-functional/commented-out CLI logic (`run_cli`).
|
||||
* `monitor.py`: Implements the automated directory monitoring feature using `watchdog`. It now processes detected archives asynchronously using a `ThreadPoolExecutor`. It utilizes `utils.prediction_utils.generate_source_rule_from_archive` for rule-based prediction and `utils.workspace_utils.prepare_processing_workspace` for workspace setup before invoking the `ProcessingEngine`.
|
||||
* `gui/`: Directory containing all code related to the Graphical User Interface (GUI), built with PySide6. The `MainWindow` acts as a coordinator, delegating functionality to specialized widgets and handlers.
|
||||
* `main_window.py`: Defines the `MainWindow` class. Acts as the main application window and coordinator, connecting signals and slots between different GUI components.
|
||||
* `main_panel_widget.py`: Defines `MainPanelWidget`, containing the primary user controls (source loading, preset selection, rule view/editor integration, processing buttons).
|
||||
* `preset_editor_widget.py`: Defines `PresetEditorWidget` for managing presets (loading, saving, editing).
|
||||
* `log_console_widget.py`: Defines `LogConsoleWidget` for displaying application logs within the GUI.
|
||||
* `rule_editor_widget.py`: Defines `RuleEditorWidget`, which houses the `QTreeView` for displaying the rule hierarchy.
|
||||
* `unified_view_model.py`: Defines `UnifiedViewModel` (`QAbstractItemModel`) for the rule hierarchy view. Holds `SourceRule` data, manages display logic (coloring), handles inline editing requests, and caches configuration data for performance.
|
||||
* `rule_hierarchy_model.py`: Defines `RuleHierarchyModel`, a simpler internal model used by `UnifiedViewModel` to manage the underlying `SourceRule` data structure.
|
||||
* `delegates.py`: Contains custom `QStyledItemDelegate` implementations used by the `UnifiedViewModel` to provide appropriate inline editors (e.g., dropdowns, text boxes) for different rule attributes.
|
||||
* `asset_restructure_handler.py`: Defines `AssetRestructureHandler`. Handles the complex logic of modifying the `SourceRule` hierarchy when a user changes a file's target asset via the GUI, ensuring data integrity. Triggered by signals from the model.
|
||||
* `base_prediction_handler.py`: Defines the abstract `BasePredictionHandler` class, providing a common interface and threading (`QRunnable`) for prediction tasks.
|
||||
* `prediction_handler.py`: Defines `RuleBasedPredictionHandler` (inherits from `BasePredictionHandler`). Generates the initial `SourceRule` hierarchy with predicted values based on input files and the selected preset rules. Runs in a background thread.
|
||||
* `llm_prediction_handler.py`: Defines `LLMPredictionHandler` (inherits from `BasePredictionHandler`). Experimental handler using an LLM for prediction. Runs in a background thread.
|
||||
* `llm_interaction_handler.py`: Defines `LLMInteractionHandler`. Manages the communication details (API calls, etc.) with the LLM service, used by `LLMPredictionHandler`.
|
||||
* `utils/`: Directory containing shared utility modules.
|
||||
* `workspace_utils.py`: Provides functions for managing processing workspaces, such as creating temporary directories and extracting archives (`prepare_processing_workspace`). Used by `main.py` (ProcessingTask) and `monitor.py`.
|
||||
* `prediction_utils.py`: Provides utility functions related to prediction, such as generating a `SourceRule` from an archive (`generate_source_rule_from_archive`), used by `monitor.py`.
|
||||
* `blenderscripts/`: Contains Python scripts (`create_nodegroups.py`, `create_materials.py`) designed to be executed *within* Blender for post-processing.
|
||||
* `Presets/`: Contains supplier-specific configuration files in JSON format.
|
||||
* `Presets/`: Contains supplier-specific configuration files in JSON format, used by the `RuleBasedPredictionHandler` for initial rule generation.
|
||||
* `Testfiles/`: Contains example input assets for testing purposes.
|
||||
* `Tickets/`: Directory for issue and feature tracking using Markdown files.
|
||||
|
||||
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `asset_processor` modules are currently being planned. The descriptions of module interactions and data flow within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
|
||||
* `Deprecated/`: Contains older code, documentation, and proof-of-concept scripts that are no longer actively used.
|
||||
@ -2,69 +2,247 @@
|
||||
|
||||
This document describes the major classes and modules that form the core of the Asset Processor Tool.
|
||||
|
||||
## `AssetProcessor` (`asset_processor.py`)
|
||||
## Core Processing Architecture
|
||||
|
||||
The `AssetProcessor` class is the central engine of the tool. It is responsible for processing a *single* input asset (either a ZIP archive or a folder) through the entire pipeline. Its key responsibilities include:
|
||||
The asset processing pipeline has been refactored into a staged architecture, managed by an orchestrator.
|
||||
|
||||
* Setting up and cleaning up a temporary workspace for processing.
|
||||
* Extracting or copying input files to the workspace.
|
||||
* Inventorying and classifying files based on configured rules (maps, models, extra, ignored, unrecognised).
|
||||
* Determining asset metadata such as name, category, and archetype.
|
||||
* Processing texture maps (resizing, format/bit depth conversion, handling Gloss->Roughness inversion, calculating statistics).
|
||||
* Merging channels from different maps according to merge rules.
|
||||
* Generating the `metadata.json` file containing details about the processed asset.
|
||||
* Organizing the final output files into the structured library directory.
|
||||
* Providing methods (`get_detailed_file_predictions`) used by the GUI for previewing file classification.
|
||||
### `ProcessingEngine` (`processing_engine.py`)
|
||||
|
||||
The `ProcessingEngine` class serves as the primary entry point for initiating an asset processing task. Its main responsibilities are:
|
||||
|
||||
* Initializing a `PipelineOrchestrator` instance.
|
||||
* Providing the `PipelineOrchestrator` with the global `Configuration` object and a predefined list of processing stages.
|
||||
* Invoking the orchestrator's `process_source_rule()` method with the input `SourceRule`, workspace path, output path, and other processing parameters.
|
||||
* Managing a top-level temporary directory for the engine's operations if needed, though individual stages might also use sub-temporary directories via the `AssetProcessingContext`.
|
||||
|
||||
It no longer contains the detailed logic for each processing step (like map manipulation, merging, etc.) directly. Instead, it delegates these tasks to the orchestrator and its stages.
|
||||
|
||||
### `PipelineOrchestrator` (`processing/pipeline/orchestrator.py`)
|
||||
|
||||
The `PipelineOrchestrator` class is responsible for managing the execution of the asset processing pipeline. Its key functions include:
|
||||
|
||||
* Receiving a `SourceRule` object, `Configuration`, and a list of `ProcessingStage` objects.
|
||||
* For each `AssetRule` within the `SourceRule`:
|
||||
* Creating an `AssetProcessingContext` instance.
|
||||
* Sequentially executing each registered `ProcessingStage`, passing the `AssetProcessingContext` to each stage.
|
||||
* Handling exceptions that occur within stages and managing the overall status of asset processing (processed, skipped, failed).
|
||||
* Managing a temporary directory for the duration of a `SourceRule` processing, which is made available to stages via the `AssetProcessingContext`.
|
||||
|
||||
### `AssetProcessingContext` (`processing/pipeline/asset_context.py`)
|
||||
|
||||
The `AssetProcessingContext` is a dataclass that acts as a stateful container for all data related to the processing of a single `AssetRule`. An instance of this context is created by the `PipelineOrchestrator` for each asset and is passed through each processing stage. Key information it holds includes:
|
||||
|
||||
* The input `SourceRule` and the current `AssetRule`.
|
||||
* Paths: `workspace_path`, `engine_temp_dir`, `output_base_path`.
|
||||
* The `Configuration` object.
|
||||
* `effective_supplier`: Determined by an early stage.
|
||||
* `asset_metadata`: A dictionary to accumulate metadata about the asset.
|
||||
* `processed_maps_details`: Stores details about individually processed maps (paths, dimensions, etc.).
|
||||
* `merged_maps_details`: Stores details about merged maps.
|
||||
* `files_to_process`: A list of `FileRule` objects to be processed for the current asset.
|
||||
* `loaded_data_cache`: For caching loaded image data within an asset's processing.
|
||||
* `status_flags`: For signaling conditions like `skip_asset` or `asset_failed`.
|
||||
* `incrementing_value`, `sha5_value`: Optional values for path generation.
|
||||
|
||||
Each stage reads from and writes to this context, allowing data and state to flow through the pipeline.
|
||||
|
||||
### `Processing Stages` (`processing/pipeline/stages/`)
|
||||
|
||||
The actual processing logic is broken down into a series of discrete stages, each inheriting from `ProcessingStage` (`processing/pipeline/stages/base_stage.py`). Each stage implements an `execute(context: AssetProcessingContext)` method. Key stages include (in typical execution order):
|
||||
|
||||
* **`SupplierDeterminationStage`**: Determines the effective supplier.
|
||||
* **`AssetSkipLogicStage`**: Checks if the asset processing should be skipped.
|
||||
* **`MetadataInitializationStage`**: Initializes basic asset metadata.
|
||||
* **`FileRuleFilterStage`**: Filters `FileRule`s to decide which files to process.
|
||||
* **`GlossToRoughConversionStage`**: Handles gloss-to-roughness map inversion.
|
||||
* **`AlphaExtractionToMaskStage`**: Extracts alpha channels to create masks.
|
||||
* **`NormalMapGreenChannelStage`**: Inverts normal map green channels if required.
|
||||
* **`IndividualMapProcessingStage`**: Processes individual maps (POT scaling, resolution variants, color conversion, stats, aspect ratio, filename conventions).
|
||||
* **`MapMergingStage`**: Merges map channels based on rules.
|
||||
* **`MetadataFinalizationAndSaveStage`**: Collects all metadata and saves `metadata.json` to a temporary location.
|
||||
* **`OutputOrganizationStage`**: Copies all processed files and metadata to the final output directory structure.
|
||||
|
||||
## `Rule Structure` (`rule_structure.py`)
|
||||
|
||||
This module defines the data structures used to represent the hierarchical processing rules:
|
||||
|
||||
* `SourceRule`: A dataclass representing rules applied at the source level. It contains nested `AssetRule` objects.
|
||||
* `AssetRule`: A dataclass representing rules applied at the asset level. It contains nested `FileRule` objects.
|
||||
* `FileRule`: A dataclass representing rules applied at the file level.
|
||||
|
||||
These classes hold specific rule parameters (e.g., `supplier_identifier`, `asset_type`, `asset_type_override`, `item_type`, `item_type_override`, `target_asset_name_override`, `resolution_override`, `channel_merge_instructions`). Attributes like `asset_type` and `item_type_override` now use string types, which are validated against centralized lists in `config/app_settings.json`. These structures support serialization (Pickle, JSON) to allow them to be passed between different parts of theapplication, including across process boundaries. The `PipelineOrchestrator` and its stages heavily rely on the information within these rule objects, passed via the `AssetProcessingContext`.
|
||||
|
||||
## `Configuration` (`configuration.py`)
|
||||
|
||||
The `Configuration` class manages the tool's settings. It is responsible for:
|
||||
|
||||
* Loading the core default settings defined in `config.py`.
|
||||
* Loading the core default settings defined in `config/app_settings.json` (e.g., `FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`, `image_resolutions`, `map_merge_rules`, `output_filename_pattern`).
|
||||
* Loading the supplier-specific rules from a selected preset JSON file (`Presets/*.json`).
|
||||
* Merging the core settings and preset rules into a single, unified configuration object.
|
||||
* Validating the loaded configuration to ensure required settings are present.
|
||||
* Pre-compiling regular expression patterns defined in the preset for efficient file classification by the `AssetProcessor`.
|
||||
* Pre-compiling regular expression patterns defined in the preset for efficient file classification by the prediction handlers.
|
||||
|
||||
An instance of the `Configuration` class is typically created once per asset processing task (within a worker process) to ensure isolated and correct settings for each asset.
|
||||
An instance of the `Configuration` class is typically created once per application run (or per processing batch) and passed to the `ProcessingEngine`, which then makes it available to the `PipelineOrchestrator` and subsequently to each stage via the `AssetProcessingContext`.
|
||||
|
||||
## `MainWindow` (`gui/main_window.py`)
|
||||
## GUI Components (`gui/`)
|
||||
|
||||
The `MainWindow` class is the main application window for the Graphical User Interface (GUI). It handles the overall UI layout and user interaction:
|
||||
The GUI has been refactored into several key components:
|
||||
|
||||
* Sets up the main window structure, including panels for the preset editor and processing controls.
|
||||
* Manages the layout of UI elements like the drag-and-drop area, preview table, buttons, and status bar.
|
||||
* Connects user actions (button clicks, drag/drop events) to corresponding handler methods (slots).
|
||||
* Interacts with background processing and prediction handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to update the UI safely from background threads.
|
||||
* Manages the GUI-specific logging handler (`QtLogHandler`) to display logs in the UI console.
|
||||
### `MainWindow` (`gui/main_window.py`)
|
||||
|
||||
## `ProcessingHandler` (`gui/processing_handler.py`)
|
||||
The `MainWindow` class acts as the main application window and **coordinator** for the GUI. Its primary responsibilities now include:
|
||||
|
||||
The `ProcessingHandler` class is designed to run in a separate `QThread` within the GUI. Its purpose is to manage the execution of the main asset processing pipeline and optional Blender scripts in the background, preventing the GUI from freezing. It:
|
||||
* Setting up the main window structure (using a `QSplitter`) and menu bar.
|
||||
* Instantiating and arranging the major GUI widgets:
|
||||
* `PresetEditorWidget` (providing selector and JSON editor parts)
|
||||
* `LLMEditorWidget` (for LLM settings)
|
||||
* `MainPanelWidget` (containing the rule view and processing controls)
|
||||
* `LogConsoleWidget`
|
||||
* **Layout Management:** Placing the preset selector statically and using a `QStackedWidget` to switch between the `PresetEditorWidget`'s JSON editor and the `LLMEditorWidget`.
|
||||
* **Editor Switching:** Handling the `preset_selection_changed_signal` from `PresetEditorWidget` to switch the stacked editor view (`_on_preset_selection_changed` slot).
|
||||
* Connecting signals and slots between widgets, models (`UnifiedViewModel`), and handlers (`LLMInteractionHandler`, `AssetRestructureHandler`).
|
||||
* Managing the overall application state related to GUI interactions (e.g., enabling/disabling controls).
|
||||
* Handling top-level actions like loading sources (drag-and-drop), initiating predictions (`update_preview`), and starting the processing task (`_on_process_requested`).
|
||||
* Managing background prediction threads (Rule-Based via `QThread`, LLM via `LLMInteractionHandler`).
|
||||
* Implementing slots (`_on_rule_hierarchy_ready`, `_on_llm_prediction_ready_from_handler`, `_on_prediction_error`, `_handle_prediction_completion`) to update the model/view when prediction results/errors arrive.
|
||||
|
||||
* Manages a `concurrent.futures.ProcessPoolExecutor` to run individual asset processing tasks (`AssetProcessor.process()`) in separate worker processes.
|
||||
* Submits processing tasks to the pool and monitors their completion.
|
||||
* Communicates progress, status updates, and results back to the `MainWindow` using Qt signals.
|
||||
* Handles the execution of Blender scripts via subprocess calls after asset processing is complete.
|
||||
* Provides logic for cancelling ongoing processing tasks (though cancellation of already running worker processes is not immediate).
|
||||
### `MainPanelWidget` (`gui/main_panel_widget.py`)
|
||||
|
||||
## `PredictionHandler` (`gui/prediction_handler.py`)
|
||||
This widget contains the central part of the GUI, including:
|
||||
|
||||
The `PredictionHandler` class also runs in a separate `QThread` in the GUI. It is responsible for generating file classification previews in the background without blocking the UI. It:
|
||||
* Controls for loading source files/directories.
|
||||
* The preset selection dropdown.
|
||||
* Buttons for initiating prediction and processing.
|
||||
* The `RuleEditorWidget` which houses the hierarchical rule view.
|
||||
|
||||
* Calls methods on the `AssetProcessor` (specifically `get_detailed_file_predictions`) to analyze input files and predict their classification and output names based on the selected processing preset.
|
||||
* Uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks.
|
||||
* Sends the prediction results back to the `MainWindow` via Qt signals to update the preview table.
|
||||
### `PresetEditorWidget` (`gui/preset_editor_widget.py`)
|
||||
|
||||
## `ZipHandler` (`monitor.py`)
|
||||
This widget provides the interface for managing presets:
|
||||
|
||||
The `ZipHandler` is a custom event handler used by the `monitor.py` script, built upon the `watchdog` library. It is responsible for:
|
||||
* Loading, saving, and editing preset files (`Presets/*.json`).
|
||||
* Displaying preset rules and settings in a tabbed JSON editor.
|
||||
* Providing the preset selection list (`QListWidget`) including the "LLM Interpretation" option.
|
||||
* **Refactored:** Exposes its selector (`selector_container`) and JSON editor (`json_editor_container`) as separate widgets for use by `MainWindow`.
|
||||
* Emits `preset_selection_changed_signal` when the selection changes.
|
||||
|
||||
* Detecting file system events, specifically the creation of new `.zip` files, in the monitored input directory.
|
||||
* Validating the filename format of detected ZIPs to extract the intended preset name.
|
||||
* Triggering the main asset processing logic (`main.run_processing`) for valid new ZIP files.
|
||||
* Managing the movement of processed source ZIP files to 'processed' or 'error' directories.
|
||||
### `LogConsoleWidget` (`gui/log_console_widget.py`)
|
||||
|
||||
These key components work together to provide the tool's functionality, separating concerns and utilizing concurrency for performance and responsiveness.
|
||||
This widget displays application logs within the GUI:
|
||||
|
||||
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of module interactions and data flow within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
|
||||
* Provides a text area for log messages.
|
||||
* Integrates with Python's `logging` system via a custom `QtLogHandler`.
|
||||
* Can be shown/hidden via the main window's "View" menu.
|
||||
|
||||
### `LLMEditorWidget` (`gui/llm_editor_widget.py`)
|
||||
|
||||
A new widget dedicated to editing LLM settings:
|
||||
|
||||
* Provides a tabbed interface ("Prompt Settings", "API Settings") to edit `config/llm_settings.json`.
|
||||
* Allows editing the main prompt, managing examples (add/delete/edit JSON), and configuring API details (URL, key, model, temperature, timeout).
|
||||
* Loads settings via `load_settings()` and saves them using `_save_settings()` (which calls `configuration.save_llm_config()`).
|
||||
* Placed within `MainWindow`'s `QStackedWidget`.
|
||||
|
||||
### `UnifiedViewModel` (`gui/unified_view_model.py`)
|
||||
|
||||
The `UnifiedViewModel` implements a `QAbstractItemModel` for use with Qt's model-view architecture. It is specifically designed to:
|
||||
|
||||
* Wrap a list of `SourceRule` objects and expose their hierarchical structure (Source -> Asset -> File) to a `QTreeView` (the Unified Hierarchical View).
|
||||
* Provide methods (`data`, `index`, `parent`, `rowCount`, `columnCount`, `flags`, `setData`) required by `QAbstractItemModel` to allow the `QTreeView` to display the rule hierarchy and support inline editing of specific attributes (e.g., `supplier_override`, `asset_type_override`, `item_type_override`, `target_asset_name_override`).
|
||||
* Handle requests for data editing (`setData`) by validating input and updating the underlying `RuleHierarchyModel`. **Note:** Complex restructuring logic (e.g., moving files between assets when `target_asset_name_override` changes) is now delegated to the `AssetRestructureHandler`.
|
||||
* Determine row background colors based on the `asset_type` and `item_type`/`item_type_override` using color metadata from the `Configuration`.
|
||||
* Hold the `SourceRule` data (via `RuleHierarchyModel`) that is the single source of truth for the GUI's processing rules.
|
||||
* Cache configuration data (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`, color maps) during initialization for improved performance in the `data()` method.
|
||||
* Includes the `update_rules_for_sources` method, which intelligently merges new prediction results into the existing model data, preserving user overrides where possible.
|
||||
|
||||
### `RuleHierarchyModel` (`gui/rule_hierarchy_model.py`)
|
||||
|
||||
A simpler, non-Qt model used internally by `UnifiedViewModel` to manage the list of `SourceRule` objects and provide methods for accessing and modifying the hierarchy.
|
||||
|
||||
### `AssetRestructureHandler` (`gui/asset_restructure_handler.py`)
|
||||
|
||||
This handler contains the complex logic required to modify the `SourceRule` hierarchy when a file's target asset is changed via the GUI's `UnifiedViewModel`. It:
|
||||
|
||||
* Is triggered by a signal (`targetAssetOverrideChanged`) from the `UnifiedViewModel`.
|
||||
* Uses dedicated methods on the `RuleHierarchyModel` (`moveFileRule`, `createAssetRule`, `removeAssetRule`) to safely move `FileRule` objects between `AssetRule`s, creating or removing `AssetRule`s as needed.
|
||||
* Ensures data consistency during these potentially complex restructuring operations.
|
||||
|
||||
### `Delegates` (`gui/delegates.py`)
|
||||
|
||||
This module contains custom `QStyledItemDelegate` implementations used by the Unified Hierarchical View (`QTreeView`) to provide inline editors for specific data types or rule attributes. Examples include delegates for:
|
||||
|
||||
* `ComboBoxDelegate`: For selecting from predefined lists of allowed asset and file types, sourced from the `Configuration` (originally from `config/app_settings.json`).
|
||||
* `LineEditDelegate`: For free-form text editing, such as the `target_asset_name_override`.
|
||||
* `SupplierSearchDelegate`: For the "Supplier" column. Provides a `QLineEdit` with auto-completion suggestions loaded from `config/suppliers.json` and handles adding/saving new suppliers.
|
||||
|
||||
These delegates handle the presentation and editing of data within the tree view cells, interacting with the `UnifiedViewModel` to get and set data.
|
||||
|
||||
## Prediction Handlers (`gui/`)
|
||||
|
||||
Prediction logic is handled by classes inheriting from a common base class, running in background threads.
|
||||
|
||||
### `BasePredictionHandler` (`gui/base_prediction_handler.py`)
|
||||
|
||||
An abstract base class (`QRunnable`) for prediction handlers. It defines the common structure and signals (`prediction_signal`) used by specific predictor implementations. It's designed to be run in a `QThreadPool`.
|
||||
|
||||
### `RuleBasedPredictionHandler` (`gui/prediction_handler.py`)
|
||||
|
||||
This class (inheriting from `BasePredictionHandler`) is responsible for generating the initial `SourceRule` hierarchy using predefined rules from presets. It:
|
||||
|
||||
* Takes an input source identifier, file list, and `Configuration` object.
|
||||
* Analyzes files based on regex patterns and rules defined in the loaded preset.
|
||||
* Constructs a `SourceRule` hierarchy with predicted values.
|
||||
* Emits the `prediction_signal` with the generated `SourceRule` object.
|
||||
|
||||
### `LLMPredictionHandler` (`gui/llm_prediction_handler.py`)
|
||||
|
||||
An experimental predictor (inheriting from `BasePredictionHandler`) that uses a Large Language Model (LLM). It:
|
||||
|
||||
* Takes an input source identifier, file list, and `Configuration` object.
|
||||
* Interacts with the `LLMInteractionHandler` to send data to the LLM and receive predictions.
|
||||
* **Parses the LLM's JSON response**: It expects a specific two-part JSON structure (see `12_LLM_Predictor_Integration.md`). It first sanitizes the response (removing comments/markdown) and then parses the JSON.
|
||||
* **Constructs `SourceRule`**: It groups files based on the `proposed_asset_group_name` from the JSON, assigns the final `asset_type` using the `asset_group_classifications` map, and builds the complete `SourceRule` hierarchy.
|
||||
* Emits the `prediction_signal` with the generated `SourceRule` object or `error_signal` on failure.
|
||||
|
||||
### `LLMInteractionHandler` (`gui/llm_interaction_handler.py`)
|
||||
|
||||
This class now acts as the central manager for LLM prediction tasks:
|
||||
|
||||
* **Manages the LLM prediction queue** and processes items sequentially.
|
||||
* **Loads LLM configuration** directly from `config/llm_settings.json` and `config/app_settings.json`.
|
||||
* **Instantiates and manages** the `LLMPredictionHandler` and its `QThread`.
|
||||
* **Handles LLM task state** (running/idle) and signals changes to the GUI.
|
||||
* Receives results/errors from `LLMPredictionHandler` and **emits signals** (`llm_prediction_ready`, `llm_prediction_error`, `llm_status_update`, `llm_processing_state_changed`) to `MainWindow`.
|
||||
|
||||
## Utility Modules (`utils/`)
|
||||
|
||||
Common utility functions have been extracted into separate modules:
|
||||
|
||||
### `workspace_utils.py`
|
||||
|
||||
Contains functions related to managing the processing workspace:
|
||||
|
||||
* `prepare_processing_workspace`: Creates temporary directories, extracts archive files (ZIP, RAR, 7z), and returns the path to the prepared workspace. Used by `main.ProcessingTask` and `monitor.py`.
|
||||
|
||||
### `prediction_utils.py`
|
||||
|
||||
Contains utility functions supporting prediction tasks:
|
||||
|
||||
* `generate_source_rule_from_archive`: A helper function used by `monitor.py` to perform rule-based prediction directly on an archive file without needing the full GUI setup. It extracts files temporarily, runs prediction logic similar to `RuleBasedPredictionHandler`, and returns a `SourceRule`.
|
||||
|
||||
## Monitor (`monitor.py`)
|
||||
|
||||
The `monitor.py` script implements the directory monitoring feature. It has been refactored to:
|
||||
|
||||
* Use `watchdog` to detect new archive files in the input directory.
|
||||
* Use a `ThreadPoolExecutor` to process detected archives asynchronously in a `_process_archive_task` function.
|
||||
* Within the task, it:
|
||||
* Loads the necessary `Configuration`.
|
||||
* Calls `utils.prediction_utils.generate_source_rule_from_archive` to get the `SourceRule`.
|
||||
* Calls `utils.workspace_utils.prepare_processing_workspace` to set up the workspace.
|
||||
* Instantiates and runs the `ProcessingEngine` (which in turn uses the `PipelineOrchestrator`).
|
||||
* Handles moving the source archive to 'processed' or 'error' directories.
|
||||
* Cleans up the workspace.
|
||||
|
||||
## Summary
|
||||
|
||||
These key components, along with the refactored GUI structure and new utility modules, work together to provide the tool's functionality. The architecture emphasizes separation of concerns (configuration, rule generation, processing, UI), utilizes background processing for responsiveness (GUI prediction, Monitor tasks), and relies on the `SourceRule` object as the central data structure passed between different stages of the workflow. The processing core is now a staged pipeline managed by the `PipelineOrchestrator`, enhancing modularity and maintainability.
|
||||
@ -2,34 +2,194 @@
|
||||
|
||||
This document provides technical details about the configuration system and the structure of preset files for developers working on the Asset Processor Tool.
|
||||
|
||||
## Configuration Flow
|
||||
## Configuration System Overview
|
||||
|
||||
The tool utilizes a two-tiered configuration system:
|
||||
The tool's configuration is managed by the `configuration.py` module and loaded from several JSON files, providing a layered approach for defaults, user overrides, definitions, and source-specific presets.
|
||||
|
||||
1. **Core Settings (`config.py`):** This Python module defines global default settings, constants, and core rules that apply generally across different asset sources. Examples include default output paths, standard image resolutions, map merge rules, output format rules, Blender executable paths, and default map types.
|
||||
2. **Preset Files (`Presets/*.json`):** These JSON files define supplier-specific rules and overrides. They contain patterns (often regular expressions) to interpret filenames, classify map types, handle variants, define naming conventions, and specify other source-specific behaviors.
|
||||
### Configuration Files
|
||||
|
||||
## `Configuration` Class (`configuration.py`)
|
||||
The tool's configuration is loaded from several JSON files, providing a layered approach for defaults, user overrides, definitions, and source-specific presets.
|
||||
|
||||
The `Configuration` class is responsible for loading, merging, and preparing the configuration settings for use by the `AssetProcessor`.
|
||||
1. **Application Settings (`config/app_settings.json`):** This JSON file defines the core global default settings, constants, and rules that apply generally across different asset sources (e.g., the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`, standard image resolutions, map merge rules, output format rules, Blender paths, temporary directory prefix, initial scaling mode, merge dimension mismatch strategy). See the [User Guide: Output Structure](../01_User_Guide/09_Output_Structure.md#available-tokens) for a list of available tokens for these patterns.
|
||||
* *Note:* `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` are no longer stored here; they have been moved to dedicated files.
|
||||
|
||||
* **Initialization:** An instance is created with a specific `preset_name`.
|
||||
* **Loading:**
|
||||
* It loads the core settings from `config.py` using `importlib.util`.
|
||||
* It loads the specified preset JSON file from the `Presets/` directory.
|
||||
* **Merging:** The loaded core settings and preset rules are merged into a single configuration object accessible via instance attributes. Preset values generally override core settings where applicable.
|
||||
* **Validation (`_validate_configs`):** Performs basic structural validation on the loaded settings, checking for the presence of required keys and basic data types (e.g., ensuring `map_type_mapping` is a list of dictionaries).
|
||||
* **Regex Compilation (`_compile_regex_patterns`):** A crucial step for performance. It iterates through the regex patterns defined in the preset (for extra files, models, bit depth variants, map keywords) and compiles them using `re.compile` (mostly case-insensitive). These compiled regex objects are stored as instance attributes (e.g., `self.compiled_map_keyword_regex`) for fast matching during file classification. It uses a helper (`_fnmatch_to_regex`) for basic wildcard (`*`, `?`) conversion in patterns.
|
||||
2. **User Settings (`config/user_settings.json`):** This optional JSON file allows users to override specific settings defined in `config/app_settings.json`. If this file exists, its values for corresponding keys will take precedence over the base application settings. This file is primarily managed through the GUI's Application Preferences Editor.
|
||||
|
||||
An instance of `Configuration` is created within each worker process (`main.process_single_asset_wrapper`) to ensure that each concurrently processed asset uses the correct, isolated configuration based on the specified preset.
|
||||
3. **Asset Type Definitions (`config/asset_type_definitions.json`):** This dedicated JSON file contains the definitions for different asset types (e.g., Surface, Model, Decal), including their descriptions, colors for UI representation, and example usage strings.
|
||||
|
||||
4. **File Type Definitions (`config/file_type_definitions.json`):** This dedicated JSON file contains the definitions for different file types (specifically texture maps and models), including descriptions, colors for UI representation, examples of keywords/patterns, a standard alias (`standard_type`), bit depth handling rules (`bit_depth_rule`), a grayscale flag (`is_grayscale`), and an optional GUI keybind (`keybind`).
|
||||
* **`keybind` Property:** Each file type object within `FILE_TYPE_DEFINITIONS` can optionally include a `keybind` property. This property accepts a single character string (e.g., `"C"`, `"R"`) representing the keyboard key. In the GUI, this key (typically combined with `Ctrl`) is used as a shortcut to set or toggle the corresponding file type for selected items in the Preview Table.
|
||||
*Example:*
|
||||
```json
|
||||
"MAP_COL": {
|
||||
"description": "Color/Albedo Map",
|
||||
"color": "#ffaa00",
|
||||
"examples": ["_col.", "_basecolor.", "albedo", "diffuse"],
|
||||
"standard_type": "COL",
|
||||
"bit_depth_rule": "force_8bit",
|
||||
"is_grayscale": false,
|
||||
"keybind": "C"
|
||||
},
|
||||
```
|
||||
Note: The `bit_depth_rule` property in `FILE_TYPE_DEFINITIONS` is the primary source for determining bit depth handling for a given map type.
|
||||
|
||||
5. **Supplier Settings (`config/suppliers.json`):** This JSON file stores settings specific to different asset suppliers. It is now structured as a dictionary where keys are supplier names and values are objects containing supplier-specific configurations.
|
||||
* **Structure:**
|
||||
```json
|
||||
{
|
||||
"SupplierName1": {
|
||||
"setting_key1": "value",
|
||||
"setting_key2": "value"
|
||||
},
|
||||
"SupplierName2": {
|
||||
"setting_key1": "value"
|
||||
}
|
||||
}
|
||||
```
|
||||
* **`normal_map_type` Property:** A key setting within each supplier's object is `normal_map_type`, specifying whether normal maps from this supplier use "OpenGL" or "DirectX" conventions.
|
||||
*Example:*
|
||||
```json
|
||||
{
|
||||
"Poliigon": {
|
||||
"normal_map_type": "DirectX"
|
||||
},
|
||||
"Dimensiva": {
|
||||
"normal_map_type": "OpenGL"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
6. **LLM Settings (`config/llm_settings.json`):** This JSON file contains settings specifically related to the LLM predictor, such as the API endpoint, model name, prompt template, and examples. These settings are managed through the GUI using the `LLMEditorWidget`.
|
||||
|
||||
7. **Preset Files (`Presets/*.json`):** These JSON files define source-specific rules and overrides. They contain patterns to interpret filenames, classify map types, handle variants, define naming conventions, and specify other source-specific behaviors. Preset settings override values from `app_settings.json` and `user_settings.json` where applicable.
|
||||
|
||||
|
||||
### Configuration Loading and Access
|
||||
|
||||
The `configuration.py` module contains the `Configuration` class and standalone functions for loading and saving settings.
|
||||
|
||||
* **`Configuration` Class:** This is the primary class used by the processing engine and other core components. When initialized with a `preset_name`, it loads settings in the following order, with later files overriding earlier ones for shared keys:
|
||||
1. `config/app_settings.json` (Base Defaults)
|
||||
2. `config/user_settings.json` (User Overrides - if exists)
|
||||
3. `config/asset_type_definitions.json` (Asset Type Definitions)
|
||||
4. `config/file_type_definitions.json` (File Type Definitions)
|
||||
5. `config/llm_settings.json` (LLM Settings)
|
||||
6. `Presets/{preset_name}.json` (Preset Overrides)
|
||||
|
||||
The loaded settings are merged into internal dictionaries, and most are accessible via instance properties (e.g., `config.output_base_dir`, `config.llm_endpoint_url`, `config.get_asset_type_definitions()`). Regex patterns defined in the merged configuration are pre-compiled for performance.
|
||||
|
||||
* **`load_base_config()` function:** This standalone function is primarily used by the GUI for initial setup and displaying default/user-overridden settings before a specific preset is selected. It loads and merges the following files:
|
||||
1. `config/app_settings.json`
|
||||
2. `config/user_settings.json` (if exists)
|
||||
3. `config/asset_type_definitions.json`
|
||||
4. `config/file_type_definitions.json`
|
||||
|
||||
It returns a single dictionary containing the combined settings and definitions.
|
||||
|
||||
* **Saving Functions:**
|
||||
* `save_base_config(settings_dict)`: Saves the provided dictionary to `config/app_settings.json`. (Used less frequently now for user-driven saves).
|
||||
* `save_user_config(settings_dict)`: Saves the provided dictionary to `config/user_settings.json`. Used by `ConfigEditorDialog`.
|
||||
* `save_llm_config(settings_dict)`: Saves the provided dictionary to `config/llm_settings.json`. Used by `LLMEditorWidget`.
|
||||
|
||||
## Supplier Management (`config/suppliers.json`)
|
||||
|
||||
A file, `config/suppliers.json`, is used to store a persistent list of known supplier names. This file is a simple JSON array of strings.
|
||||
|
||||
* **Purpose:** Provides a list of suggestions for the "Supplier" field in the GUI's Unified View, enabling auto-completion.
|
||||
* **Management:** The GUI's `SupplierSearchDelegate` is responsible for loading this list on startup, adding new, unique supplier names entered by the user, and saving the updated list back to the file.
|
||||
|
||||
## GUI Configuration Editors
|
||||
|
||||
The GUI provides dedicated editors for modifying configuration files:
|
||||
|
||||
* **`ConfigEditorDialog` (`gui/config_editor_dialog.py`):** Edits user-configurable application settings.
|
||||
* **`LLMEditorWidget` (`gui/llm_editor_widget.py`):** Edits the LLM-specific settings.
|
||||
|
||||
### `ConfigEditorDialog` (`gui/config_editor_dialog.py`)
|
||||
|
||||
The GUI includes a dedicated editor for modifying user-configurable settings. This is implemented in `gui/config_editor_dialog.py`.
|
||||
|
||||
* **Purpose:** Provides a user-friendly interface for viewing the effective application settings (defaults + user overrides + definitions) and editing the user-specific overrides.
|
||||
* **Implementation:** The dialog loads the effective settings using `load_base_config()`. It presents relevant settings in a tabbed layout ("General", "Output & Naming", etc.). When saving, it now performs a **granular save**: it loads the current content of `config/user_settings.json`, identifies only the settings that were changed by the user during the current dialog session (by comparing against the initial state), updates only those specific values in the loaded `user_settings.json` content, and saves the modified content back to `config/user_settings.json` using `save_user_config()`. This preserves any other settings in `user_settings.json` that were not touched. The dialog displays definitions from `asset_type_definitions.json` and `file_type_definitions.json` but does not save changes to these files.
|
||||
* **Limitations:** Currently, editing complex fields like `IMAGE_RESOLUTIONS` or the full details of `MAP_MERGE_RULES` via the UI is not fully supported for saving to `user_settings.json`.
|
||||
|
||||
### `LLMEditorWidget` (`gui/llm_editor_widget.py`)
|
||||
|
||||
* **Purpose:** Provides a user-friendly interface for viewing and editing the LLM settings defined in `config/llm_settings.json`.
|
||||
* **Implementation:** Uses tabs for "Prompt Settings" and "API Settings". Allows editing the prompt, managing examples, and configuring API details. When saving, it also performs a **granular save**: it loads the current content of `config/llm_settings.json`, identifies only the settings changed by the user in the current session, updates only those values, and saves the modified content back to `config/llm_settings.json` using `configuration.save_llm_config()`.
|
||||
|
||||
## Preset File Structure (`Presets/*.json`)
|
||||
|
||||
Preset files are the primary way to adapt the tool to new asset sources. Developers should use `Presets/_template.json` as a starting point. Key fields include:
|
||||
|
||||
* `supplier_name`: The name of the asset source (e.g., `"Poliigon"`). Used for output directory naming.
|
||||
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a standard internal map type (defined in `config.py`).
|
||||
* `target_type`: The standard internal map type (e.g., `"COL"`, `"NRM"`).
|
||||
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a specific file type. The `target_type` for this mapping **must** be a key from the `FILE_TYPE_DEFINITIONS` now located in `config/file_type_definitions.json`.
|
||||
* `target_type`: The specific file type key from `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_NORM_GL"`, `"MAP_RGH"`). This replaces previous alias-based systems. The common aliases like "COL" or "NRM" are now derived from the `standard_type` property within `FILE_TYPE_DEFINITIONS` but are not used directly for `target_type`.
|
||||
* `keywords`: A list of filename patterns (regex or fnmatch-style wildcards) used to identify this map type. The order of keywords within this list, and the order of dictionaries in the `map_type_mapping` list, determines the priority for assigning variant suffixes (`-1`, `-2`, etc.) when multiple files match the same `target_type`.
|
||||
* `bit_depth_variants`: A dictionary mapping standard map types (e.g., `"NRM"`) to a pattern identifying its high bit-depth variant (e.g., `"*_NRM16*.tif"`). Files matching these patterns are prioritized over their standard counterparts.
|
||||
* `map_bit_depth_rules`: Defines how to handle the bit depth of source maps. Can specify a default behavior (`"respect"` or `"force_8bit"`) and overrides for specific map types.
|
||||
* `model_patterns`: A list of regex patterns to identify model files (e.g., `".*\\.fbx"`, `".*\\.obj"`).
|
||||
* `move_to_extra_patterns`: A list of regex patterns for files that should be moved directly to the `Extra/` output subdirectory without further processing.
|
||||
* `source_naming_convention`: Rules for extracting the base asset name and potentially the archetype from source filenames or directory structures (e.g., using separators and indices).
|
||||
* `asset_category_rules`: Keywords or patterns used to determine the asset category (e.g., identifying `"Decal"` based on keywords).
|
||||
* `archetype_rules`: Keywords or patterns used to determine the asset archetype (e.g., identifying `"Wood"` or `"Metal"`).
|
||||
|
||||
Careful definition of these patterns and rules, especially the regex in `map_type_mapping`, `bit_depth_variants`, `model_patterns`, and `move_to_extra_patterns`, is essential for correct asset processing.
|
||||
|
||||
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how configuration data is handled and passed within this document reflect the current state and will require review and updates once the plan for these changes is finalized.
|
||||
|
||||
## Supplier Management (`config/suppliers.json`)
|
||||
|
||||
A new file, `config/suppliers.json`, is used to store a persistent list of known supplier names. This file is a simple JSON array of strings.
|
||||
|
||||
* **Purpose:** Provides a list of suggestions for the "Supplier" field in the GUI's Unified View, enabling auto-completion.
|
||||
* **Management:** The GUI's `SupplierSearchDelegate` is responsible for loading this list on startup, adding new, unique supplier names entered by the user, and saving the updated list back to the file.
|
||||
|
||||
## `Configuration` Class (`configuration.py`)
|
||||
|
||||
The `Configuration` class is central to the new configuration system. It is responsible for loading, merging, and preparing the configuration settings for use by the `ProcessingEngine` and other components like the `PredictionHandler` and `LLMPredictionHandler`.
|
||||
|
||||
* **Initialization:** An instance is created with a specific `preset_name`.
|
||||
* **Loading:**
|
||||
* It first loads the base application settings from `config/app_settings.json`.
|
||||
* It then loads the LLM-specific settings from `config/llm_settings.json`.
|
||||
* Finally, it loads the specified preset JSON file from the `Presets/` directory.
|
||||
* **Merging & Access:** The base settings from `app_settings.json` are merged with the preset rules. LLM settings are stored separately. Most settings are accessible via instance properties (e.g., `config.llm_endpoint_url`). Preset values generally override the base settings where applicable. **Exception:** The `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN` are loaded *only* from `app_settings.json` and are accessed via `config.output_directory_pattern` and `config.output_filename_pattern` respectively; they are not defined in or overridden by presets.
|
||||
* **Validation (`_validate_configs`):** Performs basic structural validation on the loaded settings (base, LLM, and preset), checking for the presence of required keys and basic data types. Logs warnings for missing optional LLM keys.
|
||||
* **Regex Compilation (`_compile_regex_patterns`):** Compiles regex patterns defined in the merged configuration (from base settings and the preset) for performance. Compiled regex objects are stored as instance attributes (e.g., `self.compiled_map_keyword_regex`).
|
||||
* **LLM Settings Access:** The `Configuration` class provides direct property access (e.g., `config.llm_endpoint_url`, `config.llm_api_key`, `config.llm_model_name`, `config.llm_temperature`, `config.llm_request_timeout`, `config.llm_predictor_prompt`, `config.get_llm_examples()`) to allow components like the `LLMPredictionHandler` to easily access the necessary LLM configuration values loaded from `config/llm_settings.json`.
|
||||
|
||||
An instance of `Configuration` is created within each worker process (`main.process_single_asset_wrapper`) to ensure that each concurrently processed asset uses the correct, isolated configuration based on the specified preset and the base application settings. The `LLMInteractionHandler` loads LLM settings directly using helper functions or file access, not the `Configuration` class.
|
||||
|
||||
## GUI Configuration Editors
|
||||
|
||||
The GUI provides dedicated editors for modifying configuration files:
|
||||
|
||||
* **`ConfigEditorDialog` (`gui/config_editor_dialog.py`):** Edits the core `config/app_settings.json`.
|
||||
* **`LLMEditorWidget` (`gui/llm_editor_widget.py`):** Edits the LLM-specific `config/llm_settings.json`.
|
||||
|
||||
### `ConfigEditorDialog` (`gui/config_editor_dialog.py`)
|
||||
|
||||
The GUI includes a dedicated editor for modifying the `config/app_settings.json` file. This is implemented in `gui/config_editor_dialog.py`.
|
||||
|
||||
* **Purpose:** Provides a user-friendly interface for viewing and editing the core application settings defined in `app_settings.json`.
|
||||
* **Implementation:** The dialog loads the JSON content of `app_settings.json`, presents it in a tabbed layout ("General", "Output & Naming", etc.) using standard GUI widgets mapped to the JSON structure, and saves the changes back to the file. The "Output & Naming" tab specifically handles the global `OUTPUT_DIRECTORY_PATTERN` and `OUTPUT_FILENAME_PATTERN`. It supports editing basic fields, tables for definitions (`FILE_TYPE_DEFINITIONS`, `ASSET_TYPE_DEFINITIONS`), and a list/detail view for merge rules (`MAP_MERGE_RULES`). The definitions tables include dynamic color editing features.
|
||||
* **Limitations:** Currently, editing complex fields like `IMAGE_RESOLUTIONS` or the full details of `MAP_MERGE_RULES` via the UI is not fully supported.
|
||||
* **Note:** Changes made through the `ConfigEditorDialog` are written directly to `config/app_settings.json` (using `save_base_config`) but require an application restart to be loaded and applied by the `Configuration` class during processing.
|
||||
|
||||
### `LLMEditorWidget` (`gui/llm_editor_widget.py`)
|
||||
|
||||
* **Purpose:** Provides a user-friendly interface for viewing and editing the LLM settings defined in `config/llm_settings.json`.
|
||||
* **Implementation:** Uses tabs for "Prompt Settings" and "API Settings". Allows editing the prompt, managing examples, and configuring API details.
|
||||
* **Persistence:** Saves changes directly to `config/llm_settings.json` using the `configuration.save_llm_config()` function. Changes are loaded by the `LLMInteractionHandler` the next time an LLM task is initiated.
|
||||
|
||||
## Preset File Structure (`Presets/*.json`)
|
||||
|
||||
Preset files are the primary way to adapt the tool to new asset sources. Developers should use `Presets/_template.json` as a starting point. Key fields include:
|
||||
|
||||
* `supplier_name`: The name of the asset source (e.g., `"Poliigon"`). Used for output directory naming.
|
||||
* `map_type_mapping`: A list of dictionaries, each mapping source filename patterns/keywords to a specific file type. The `target_type` for this mapping **must** be a key from `FILE_TYPE_DEFINITIONS` located in `config/app_settings.json`.
|
||||
* `target_type`: The specific file type key from `FILE_TYPE_DEFINITIONS` (e.g., `"MAP_COL"`, `"MAP_NORM_GL"`, `"MAP_RGH"`). This replaces previous alias-based systems. The common aliases like "COL" or "NRM" are now derived from the `standard_type` property within `FILE_TYPE_DEFINITIONS` but are not used directly for `target_type`.
|
||||
* `keywords`: A list of filename patterns (regex or fnmatch-style wildcards) used to identify this map type. The order of keywords within this list, and the order of dictionaries in the `map_type_mapping` list, determines the priority for assigning variant suffixes (`-1`, `-2`, etc.) when multiple files match the same `target_type`.
|
||||
* `bit_depth_variants`: A dictionary mapping standard map types (e.g., `"NRM"`) to a pattern identifying its high bit-depth variant (e.g., `"*_NRM16*.tif"`). Files matching these patterns are prioritized over their standard counterparts.
|
||||
* `map_bit_depth_rules`: Defines how to handle the bit depth of source maps. Can specify a default behavior (`"respect"` or `"force_8bit"`) and overrides for specific map types.
|
||||
|
||||
@ -1,82 +1,98 @@
|
||||
# Developer Guide: Processing Pipeline
|
||||
Cl# Developer Guide: Processing Pipeline
|
||||
|
||||
This document details the step-by-step technical process executed by the `AssetProcessor` class (`asset_processor.py`) when processing a single asset.
|
||||
This document details the step-by-step technical process executed by the asset processing pipeline, which is initiated by the [`ProcessingEngine`](processing_engine.py:73) class (`processing_engine.py`) and orchestrated by the [`PipelineOrchestrator`](processing/pipeline/orchestrator.py:36) (`processing/pipeline/orchestrator.py`).
|
||||
|
||||
The `AssetProcessor.process()` method orchestrates the following pipeline:
|
||||
The [`ProcessingEngine.process()`](processing_engine.py:131) method serves as the main entry point. It initializes a [`PipelineOrchestrator`](processing/pipeline/orchestrator.py:36) instance, providing it with the application's [`Configuration`](configuration.py:68) object and predefined lists of pre-item and post-item processing stages. The [`PipelineOrchestrator.process_source_rule()`](processing/pipeline/orchestrator.py:95) method then manages the execution of these stages for each asset defined in the input [`SourceRule`](rule_structure.py:40).
|
||||
|
||||
1. **Workspace Setup (`_setup_workspace`)**:
|
||||
* Creates a temporary directory using `tempfile.mkdtemp()` to isolate the processing of the current asset.
|
||||
A crucial component in this architecture is the [`AssetProcessingContext`](processing/pipeline/asset_context.py:86) (`processing/pipeline/asset_context.py`). An instance of this dataclass is created for each [`AssetRule`](rule_structure.py:22) being processed. It acts as a stateful container, carrying all relevant data (source files, rules, configuration, intermediate results, metadata) and is passed sequentially through each stage. Each stage can read from and write to the context, allowing data to flow and be modified throughout the pipeline.
|
||||
|
||||
2. **Input Extraction (`_extract_input`)**:
|
||||
* If the input is a supported archive type (.zip, .rar, .7z), it's extracted into the temporary workspace using the appropriate library (`zipfile`, `rarfile`, or `py7zr`).
|
||||
* If the input is a directory, its contents are copied into the temporary workspace.
|
||||
* Includes basic error handling for invalid or password-protected archives.
|
||||
The pipeline execution for each asset follows this general flow:
|
||||
|
||||
3. **File Inventory and Classification (`_inventory_and_classify_files`)**:
|
||||
* Scans the contents of the temporary workspace.
|
||||
* Uses the pre-compiled regex patterns from the loaded `Configuration` object to classify each file.
|
||||
* Classification follows a multi-pass approach for priority:
|
||||
* Explicitly marked `Extra/` files (using `move_to_extra_patterns` regex).
|
||||
* Model files (using `model_patterns` regex).
|
||||
* Potential Texture Maps (matching `map_type_mapping` keyword patterns).
|
||||
* Standalone 16-bit variants check (using `bit_depth_variants` patterns).
|
||||
* Prioritization of 16-bit variants over their 8-bit counterparts (marking the 8-bit version as `Ignored`).
|
||||
* Final classification of remaining potential maps.
|
||||
* Remaining files are classified as `Unrecognised` (and typically moved to `Extra/` later).
|
||||
* Stores the classification results (including source path, determined map type, potential variant suffix, etc.) in `self.classified_files`.
|
||||
* Sorts potential map variants based on preset rule order, keyword order within the rule, and finally alphabetical path to determine suffix assignment priority (`-1`, `-2`, etc.).
|
||||
1. **Pre-Item Stages:** A sequence of stages executed once per asset before the core item processing loop. These stages typically perform initial setup, filtering, and asset-level transformations.
|
||||
2. **Core Item Processing Loop:** The [`PipelineOrchestrator`](processing/pipeline/orchestrator.py:36) iterates through a list of "processing items" (individual files or merge tasks) prepared by a dedicated stage. For each item, a sequence of core processing stages is executed.
|
||||
3. **Post-Item Stages:** A sequence of stages executed once per asset after the core item processing loop is complete. These stages handle final tasks like organizing output files and saving metadata.
|
||||
|
||||
4. **Base Metadata Determination (`_determine_base_metadata`, `_determine_single_asset_metadata`)**:
|
||||
* Determines the base asset name using `source_naming_convention` rules from the `Configuration` (separators, indices), with fallbacks to common prefixes or the input name. Handles multiple distinct assets within a single input source.
|
||||
* Determines the asset category (`Texture`, `Asset`, `Decal`) based on the presence of model files or `decal_keywords` in the `Configuration`.
|
||||
* Determines the asset archetype (e.g., `Wood`, `Metal`) by matching keywords from `archetype_rules` (in `Configuration`) against file stems or the determined base name.
|
||||
* Stores this preliminary metadata.
|
||||
## Pipeline Stages
|
||||
|
||||
5. **Skip Check**:
|
||||
* If the `overwrite` flag (passed during initialization) is `False`, the tool checks if the final output directory for the determined asset name already exists and contains a `metadata.json` file.
|
||||
* If both exist, processing for this specific asset is skipped, marked as "skipped", and the pipeline moves to the next asset (if processing multiple assets from one source) or finishes.
|
||||
The stages are executed in the following order for each asset:
|
||||
|
||||
6. **Map Processing (`_process_maps`)**:
|
||||
* Iterates through the files classified as texture maps for the current asset.
|
||||
* Loads the image using `cv2.imread` (handling grayscale and unchanged flags). Converts BGR to RGB internally for consistency (except for saving non-EXR formats).
|
||||
* Handles Glossiness-to-Roughness inversion if necessary (loads gloss, inverts `1.0 - img/norm`, prioritizes gloss source if both exist).
|
||||
* Resizes the image to target resolutions defined in `IMAGE_RESOULTIONS` (from `Configuration`) using `cv2.resize` (`INTER_LANCZOS4` for downscaling). Upscaling is generally avoided by checks.
|
||||
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` vs `force_8bit`).
|
||||
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a hierarchy of rules:
|
||||
* `FORCE_LOSSLESS_MAP_TYPES` list (overrides other logic).
|
||||
* `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for large 8-bit maps).
|
||||
* Source format, target bit depth, and configured defaults (`OUTPUT_FORMAT_16BIT_PRIMARY`, `OUTPUT_FORMAT_8BIT`).
|
||||
* Converts the NumPy array data type appropriately before saving (e.g., float to uint8/uint16 with scaling).
|
||||
* Saves the processed map using `cv2.imwrite` (converting RGB back to BGR if saving to non-EXR formats). Includes fallback logic (e.g., attempting PNG if saving 16-bit EXR fails).
|
||||
* Calculates image statistics (Min/Max/Mean) using `_calculate_image_stats` on normalized float64 data for the `CALCULATE_STATS_RESOLUTION`.
|
||||
* Determines the aspect ratio change string (e.g., `"EVEN"`, `"X150"`) using `_normalize_aspect_ratio_change`.
|
||||
* Stores details about each processed map (path, resolution, format, stats, etc.) in `processed_maps_details_asset`.
|
||||
### Pre-Item Stages
|
||||
|
||||
7. **Map Merging (`_merge_maps_from_source`)**:
|
||||
* Iterates through the `MAP_MERGE_RULES` defined in the `Configuration`.
|
||||
* Identifies the required *source* map files needed as input for each merge rule based on the classified files.
|
||||
* Determines common resolutions available across the required input maps.
|
||||
* Loads the necessary source map channels for each common resolution (using a helper `_load_and_transform_source` which includes caching).
|
||||
* Converts inputs to normalized float32 (0-1).
|
||||
* Injects default channel values (from rule `defaults`) if an input channel is missing.
|
||||
* Merges channels using `cv2.merge`.
|
||||
* Determines output bit depth and format based on rules (similar logic to `_process_maps`, considering input properties). Handles potential JPG 16-bit conflict by forcing 8-bit.
|
||||
* Saves the merged map using the `_save_image` helper (includes data type/color space conversions and fallback).
|
||||
* Stores details about each merged map in `merged_maps_details_asset`.
|
||||
These stages are executed sequentially once for each asset before the core item processing loop begins.
|
||||
|
||||
8. **Metadata File Generation (`_generate_metadata_file`)**:
|
||||
* Collects all determined information for the current asset: base metadata, details from `processed_maps_details_asset` and `merged_maps_details_asset`, list of ignored files, source preset used, etc.
|
||||
* Writes this collected data into the `metadata.json` file within the temporary workspace using `json.dump`.
|
||||
1. **[`SupplierDeterminationStage`](processing/pipeline/stages/supplier_determination.py:6)** (`processing/pipeline/stages/supplier_determination.py`):
|
||||
* **Responsibility**: Determines the effective supplier for the asset based on the [`SourceRule`](rule_structure.py:40)'s `supplier_override`, `supplier_identifier`, and validation against configured suppliers.
|
||||
* **Context Interaction**: Sets `context.effective_supplier` and may set a `supplier_error` flag in `context.status_flags`.
|
||||
|
||||
9. **Output Organization (`_organize_output_files`)**:
|
||||
* Creates the final structured output directory: `<output_base_dir>/<supplier_name>/<asset_name>/`.
|
||||
* Creates subdirectories `Extra/`, `Unrecognised/`, and `Ignored/` within the asset directory.
|
||||
* Moves the processed maps, merged maps, model files, `metadata.json`, and files classified as Extra, Unrecognised, or Ignored from the temporary workspace into their respective locations in the final output directory structure.
|
||||
2. **[`AssetSkipLogicStage`](processing/pipeline/stages/asset_skip_logic.py:5)** (`processing/pipeline/stages/asset_skip_logic.py`):
|
||||
* **Responsibility**: Checks if the entire asset should be skipped based on conditions like a missing/invalid supplier, a "SKIP" status in asset metadata, or if the asset is already processed and overwrite is disabled.
|
||||
* **Context Interaction**: Sets the `skip_asset` flag and `skip_reason` in `context.status_flags` if the asset should be skipped.
|
||||
|
||||
10. **Workspace Cleanup (`_cleanup_workspace`)**:
|
||||
* Removes the temporary workspace directory and its contents using `shutil.rmtree()`. This is called within a `finally` block to ensure cleanup is attempted even if errors occur during processing.
|
||||
3. **[`MetadataInitializationStage`](processing/pipeline/stages/metadata_initialization.py:81)** (`processing/pipeline/stages/metadata_initialization.py`):
|
||||
* **Responsibility**: Initializes the `context.asset_metadata` dictionary with base information derived from the [`AssetRule`](rule_structure.py:22), [`SourceRule`](rule_structure.py:40), and [`Configuration`](configuration.py:68). This includes asset name, IDs, source/output paths, timestamps, and initial status.
|
||||
* **Context Interaction**: Populates `context.asset_metadata`. Initializes `context.processed_maps_details` and `context.merged_maps_details` as empty dictionaries (these are used internally by subsequent stages but are not directly part of the final `metadata.json` in their original form).
|
||||
|
||||
11. **(Optional) Blender Script Execution**:
|
||||
* If triggered via CLI arguments (`--nodegroup-blend`, `--materials-blend`) or GUI controls, the orchestrator (`main.py` or `gui/processing_handler.py`) executes the corresponding Blender scripts (`blenderscripts/*.py`) using `subprocess.run` after the `AssetProcessor.process()` call completes successfully for an asset batch. See `Developer Guide: Blender Integration Internals` for more details.
|
||||
4. **[`FileRuleFilterStage`](processing/pipeline/stages/file_rule_filter.py:10)** (`processing/pipeline/stages/file_rule_filter.py`):
|
||||
* **Responsibility**: Filters the [`FileRule`](rule_structure.py:5) objects associated with the asset to determine which individual files should be considered for processing. It identifies and excludes files matching "FILE_IGNORE" rules based on their `item_type`.
|
||||
* **Context Interaction**: Populates `context.files_to_process` with the list of [`FileRule`](rule_structure.py:5) objects that are not ignored.
|
||||
|
||||
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how data is processed and transformed within this pipeline reflect the current state and will require review and updates once the plan for these changes is finalized.
|
||||
5. **[`GlossToRoughConversionStage`](processing/pipeline/stages/gloss_to_rough_conversion.py:15)** (`processing/pipeline/stages/gloss_to_rough_conversion.py`):
|
||||
* **Responsibility**: Identifies processed maps in `context.processed_maps_details` whose `internal_map_type` starts with "MAP_GLOSS". If found, it loads the temporary image data, inverts it using the shared utility function [`apply_common_map_transformations`](processing/utils/image_processing_utils.py), saves a new temporary roughness map ("MAP_ROUGH"), and updates the corresponding details in `context.processed_maps_details` (setting `internal_map_type` to "MAP_ROUGH") and the relevant [`FileRule`](rule_structure.py:5) in `context.files_to_process` (setting `item_type` to "MAP_ROUGH").
|
||||
* **Context Interaction**: Reads from and updates `context.processed_maps_details` (specifically `internal_map_type` and `temp_processed_file`) and `context.files_to_process` (specifically `item_type`).
|
||||
|
||||
6. **[`AlphaExtractionToMaskStage`](processing/pipeline/stages/alpha_extraction_to_mask.py:16)** (`processing/pipeline/stages/alpha_extraction_to_mask.py`):
|
||||
* **Responsibility**: If no mask map is explicitly defined for the asset (as a [`FileRule`](rule_structure.py:5) with `item_type="MAP_MASK"`), this stage searches `context.processed_maps_details` for a suitable source map (e.g., a "MAP_COL" with an alpha channel, based on its `internal_map_type`). If found, it extracts the alpha channel, saves it as a new temporary mask map, and adds a new [`FileRule`](rule_structure.py:5) (with `item_type="MAP_MASK"`) and corresponding details (with `internal_map_type="MAP_MASK"`) to the context.
|
||||
* **Context Interaction**: Reads from `context.processed_maps_details`, adds a new [`FileRule`](rule_structure.py:5) to `context.files_to_process`, and adds a new entry to `context.processed_maps_details` (setting `internal_map_type`).
|
||||
|
||||
7. **[`NormalMapGreenChannelStage`](processing/pipeline/stages/normal_map_green_channel.py:14)** (`processing/pipeline/stages/normal_map_green_channel.py`):
|
||||
* **Responsibility**: Identifies processed normal maps in `context.processed_maps_details` (those with an `internal_map_type` starting with "MAP_NRM"). If the global `invert_normal_map_green_channel_globally` configuration is true, it loads the temporary image data, inverts the green channel using the shared utility function [`apply_common_map_transformations`](processing/utils/image_processing_utils.py), saves a new temporary modified normal map, and updates the `temp_processed_file` path in `context.processed_maps_details`.
|
||||
* **Context Interaction**: Reads from and updates `context.processed_maps_details` (specifically `temp_processed_file` and `notes`).
|
||||
|
||||
### Core Item Processing Loop
|
||||
|
||||
The [`PipelineOrchestrator`](processing/pipeline/orchestrator.py:36) iterates through the `context.processing_items` list (populated by the [`PrepareProcessingItemsStage`](processing/pipeline/stages/prepare_processing_items.py:10)). For each item (either a [`FileRule`](rule_structure.py:5) for a regular map or a [`MergeTaskDefinition`](processing/pipeline/asset_context.py:16) for a merged map), the following stages are executed sequentially:
|
||||
|
||||
1. **[`PrepareProcessingItemsStage`](processing/pipeline/stages/prepare_processing_items.py:10)** (`processing/pipeline/stages/prepare_processing_items.py`):
|
||||
* **Responsibility**: (Executed once before the loop) Creates the `context.processing_items` list by combining [`FileRule`](rule_structure.py:5)s from `context.files_to_process` and [`MergeTaskDefinition`](processing/pipeline/asset_context.py:16)s derived from the global `map_merge_rules` configuration. It correctly accesses `map_merge_rules` from `context.config_obj` and validates each merge rule for the presence of `output_map_type` and a dictionary for `inputs`. Initializes `context.intermediate_results`.
|
||||
* **Context Interaction**: Reads from `context.files_to_process` and `context.config_obj` (accessing `map_merge_rules`). Populates `context.processing_items` and initializes `context.intermediate_results`.
|
||||
|
||||
2. **[`RegularMapProcessorStage`](processing/pipeline/stages/regular_map_processor.py:18)** (`processing/pipeline/stages/regular_map_processor.py`):
|
||||
* **Responsibility**: (Executed per [`FileRule`](rule_structure.py:5) item) Checks if the `FileRule.item_type` starts with "MAP_". If not, the item is skipped. Otherwise, it loads the image data for the file, determines its potentially suffixed internal map type (e.g., "MAP_COL-1"), applies in-memory transformations (Gloss-to-Rough, Normal Green Invert) using the shared utility function [`apply_common_map_transformations`](processing/utils/image_processing_utils.py), and returns the processed image data and details in a [`ProcessedRegularMapData`](processing/pipeline/asset_context.py:23) object. The `internal_map_type` in the output reflects any transformations (e.g., "MAP_GLOSS" becomes "MAP_ROUGH").
|
||||
* **Context Interaction**: Reads from the input [`FileRule`](rule_structure.py:5) (checking `item_type`) and [`Configuration`](configuration.py:68). Returns a [`ProcessedRegularMapData`](processing/pipeline/asset_context.py:23) object which is stored in `context.intermediate_results`.
|
||||
|
||||
3. **[`MergedTaskProcessorStage`](processing/pipeline/stages/merged_task_processor.py:68)** (`processing/pipeline/stages/merged_task_processor.py`):
|
||||
* **Responsibility**: (Executed per [`MergeTaskDefinition`](processing/pipeline/asset_context.py:16) item) Validates that all input map types specified in the merge rule start with "MAP_". If not, the task is failed. It dynamically loads input images by looking up the required input map types (e.g., "MAP_NRM") in `context.processed_maps_details` and using the temporary file paths from their `saved_files_info`. It applies in-memory transformations to inputs using [`apply_common_map_transformations`](processing/utils/image_processing_utils.py), handles dimension mismatches (with fallback creation if configured and `source_dimensions` are available), performs the channel merging operation, and returns the merged image data and details in a [`ProcessedMergedMapData`](processing/pipeline/asset_context.py:35) object. The `output_map_type` of the merged map must also be "MAP_" prefixed in the configuration.
|
||||
* **Context Interaction**: Reads from the input [`MergeTaskDefinition`](processing/pipeline/asset_context.py:16) (checking input map types), `context.workspace_path`, `context.processed_maps_details` (for input image data), and [`Configuration`](configuration.py:68). Returns a [`ProcessedMergedMapData`](processing/pipeline/asset_context.py:35) object which is stored in `context.intermediate_results`.
|
||||
|
||||
4. **[`InitialScalingStage`](processing/pipeline/stages/initial_scaling.py:14)** (`processing/pipeline/stages/initial_scaling.py`):
|
||||
* **Responsibility**: (Executed per item) Applies initial scaling (e.g., Power-of-Two downscaling) to the image data from the previous processing stage based on the `initial_scaling_mode` configuration.
|
||||
* **Context Interaction**: Takes a [`InitialScalingInput`](processing/pipeline/asset_context.py:46) (containing image data and config) and returns an [`InitialScalingOutput`](processing/pipeline/asset_context.py:54) object, which updates the item's entry in `context.intermediate_results`.
|
||||
|
||||
5. **[`SaveVariantsStage`](processing/pipeline/stages/save_variants.py:15)** (`processing/pipeline/stages/save_variants.py`):
|
||||
* **Responsibility**: (Executed per item) Takes the final processed image data (potentially scaled) and configuration, and calls a utility to save the image to temporary files in various resolutions and formats as defined by the configuration.
|
||||
* **Context Interaction**: Takes a [`SaveVariantsInput`](processing/pipeline/asset_context.py:61) object (which includes the "MAP_" prefixed `internal_map_type`). It uses the `get_filename_friendly_map_type` utility to convert this to a "standard type" (e.g., "COL") for output naming. Returns a [`SaveVariantsOutput`](processing/pipeline/asset_context.py:79) object containing details about the saved temporary files. The orchestrator stores these details, including the original "MAP_" prefixed `internal_map_type`, in `context.processed_maps_details` for the item.
|
||||
|
||||
### Post-Item Stages
|
||||
|
||||
These stages are executed sequentially once for each asset after the core item processing loop has finished for all items.
|
||||
|
||||
1. **[`OutputOrganizationStage`](processing/pipeline/stages/output_organization.py:14)** (`processing/pipeline/stages/output_organization.py`):
|
||||
* **Responsibility**: Determines the final output paths for all processed maps (including variants) and extra files based on configured patterns. It copies the temporary files generated by the core stages to these final destinations, creating directories as needed and respecting overwrite settings.
|
||||
* **Context Interaction**: Reads from `context.processed_maps_details`, `context.files_to_process` (for 'EXTRA' files), `context.output_base_path`, and [`Configuration`](configuration.py:68). Updates entries in `context.processed_maps_details` with organization status. Populates `context.asset_metadata['maps']` with the final map structure:
|
||||
* The `maps` object is a dictionary where keys are standard map types (e.g., "COL", "REFL").
|
||||
* Each entry contains a `variant_paths` dictionary, where keys are resolution strings (e.g., "8K", "4K") and values are the filenames of the map variants (relative to the asset's output directory).
|
||||
It also populates `context.asset_metadata['final_output_files']` with a list of absolute paths to all generated files (this list itself is not saved in the final `metadata.json`).
|
||||
|
||||
2. **[`MetadataFinalizationAndSaveStage`](processing/pipeline/stages/metadata_finalization_save.py:14)** (`processing/pipeline/stages/metadata_finalization_save.py`):
|
||||
* **Responsibility**: Finalizes the `context.asset_metadata` (setting final status based on flags). It determines the save path for the metadata file based on configuration and patterns, serializes the `context.asset_metadata` (which now contains the structured `maps` data from `OutputOrganizationStage`) to JSON, and saves the `metadata.json` file.
|
||||
* **Context Interaction**: Reads from `context.asset_metadata` (including the `maps` structure), `context.output_base_path`, and [`Configuration`](configuration.py:68). Before saving, it explicitly removes the `final_output_files` key from `context.asset_metadata`. The `processing_end_time` is also no longer added. The `metadata.json` file is written, and `context.asset_metadata` is updated with its final path and status. The older `processed_maps_details` and `merged_maps_details` from the context are not directly included in the JSON.
|
||||
|
||||
## External Steps
|
||||
|
||||
Certain steps are integral to the overall asset processing workflow but are handled outside the [`PipelineOrchestrator`](processing/pipeline/orchestrator.py:36)'s direct execution loop:
|
||||
|
||||
* **Workspace Preparation and Cleanup**: Handled by the code that invokes [`ProcessingEngine.process()`](processing_engine.py:131) (e.g., `main.ProcessingTask`, `monitor._process_archive_task`), typically involving extracting archives and setting up temporary directories. The engine itself manages a sub-temporary directory (`engine_temp_dir`) for intermediate processing files.
|
||||
* **Prediction and Rule Generation**: Performed before the [`ProcessingEngine`](processing_engine.py:73) is called. This involves analyzing source files and generating the [`SourceRule`](rule_structure.py:40) object with its nested [`AssetRule`](rule_structure.py:22)s and [`FileRule`](rule_structure.py:5)s, often involving prediction logic (potentially using LLMs).
|
||||
* **Optional Blender Script Execution**: Can be triggered externally after successful processing to perform tasks like material setup in Blender using the generated output files and metadata.
|
||||
|
||||
This staged pipeline provides a modular and extensible architecture for asset processing, with clear separation of concerns for each step. The [`AssetProcessingContext`](processing/pipeline/asset_context.py:86) ensures that data flows consistently between these stages.
|
||||
@ -8,98 +8,214 @@ The GUI is built using `PySide6`, which provides Python bindings for the Qt fram
|
||||
|
||||
## Main Window (`gui/main_window.py`)
|
||||
|
||||
The `MainWindow` class is the central component of the GUI application. It is responsible for:
|
||||
The `MainWindow` class acts as the central **coordinator** for the GUI application. It is responsible for:
|
||||
|
||||
* Defining the main application window structure and layout using PySide6 widgets.
|
||||
* Arranging the Preset Editor panel (left) and the Processing panel (right).
|
||||
* Setting up the menu bar, including the "View" menu for toggling the Log Console and Detailed File Preview.
|
||||
* Connecting user interactions (button clicks, drag-and-drop events, checkbox states, spinbox values) to corresponding methods (slots) within the `MainWindow` or other handler classes.
|
||||
* Managing the display of application logs in the UI console using a custom `QtLogHandler`.
|
||||
* Interacting with background handlers (`ProcessingHandler`, `PredictionHandler`) via Qt signals and slots to ensure thread-safe updates to the UI during long-running operations.
|
||||
* Setting up the main application window structure and menu bar, including actions to launch configuration and definition editors.
|
||||
* **Layout:** Arranging the main GUI components using a `QSplitter`.
|
||||
* **Left Pane:** Contains the preset selection controls (from `PresetEditorWidget`) permanently displayed at the top. Below this, a `QStackedWidget` switches between the preset JSON editor (also from `PresetEditorWidget`) and the `LLMEditorWidget`.
|
||||
* **Right Pane:** Contains the `MainPanelWidget`.
|
||||
* Instantiating and managing the major GUI widgets:
|
||||
* `PresetEditorWidget` (`gui/preset_editor_widget.py`): Provides the preset selector and the JSON editor parts.
|
||||
* `LLMEditorWidget` (`gui/llm_editor_widget.py`): Provides the editor for LLM settings (from `config/llm_settings.json`).
|
||||
* `MainPanelWidget` (`gui/main_panel_widget.py`): Contains the rule hierarchy view and processing controls.
|
||||
* `LogConsoleWidget` (`gui/log_console_widget.py`): Displays application logs.
|
||||
* Instantiating key models and handlers:
|
||||
* `UnifiedViewModel` (`gui/unified_view_model.py`): The model for the rule hierarchy view.
|
||||
* `LLMInteractionHandler` (`gui/llm_interaction_handler.py`): Manages communication with the LLM service.
|
||||
* `AssetRestructureHandler` (`gui/asset_restructure_handler.py`): Handles rule restructuring.
|
||||
* Connecting signals and slots between these components to orchestrate the application flow.
|
||||
* **Editor Switching:** Handling the `preset_selection_changed_signal` from `PresetEditorWidget` in its `_on_preset_selection_changed` slot. This slot:
|
||||
* Switches the `QStackedWidget` (`editor_stack`) to display either the `PresetEditorWidget`'s JSON editor or the `LLMEditorWidget` based on the selected mode ("preset", "llm", "placeholder").
|
||||
* Calls `llm_editor_widget.load_settings()` when switching to LLM mode.
|
||||
* Updates the window title.
|
||||
* Triggers `update_preview()`.
|
||||
* Handling top-level user interactions like drag-and-drop for loading sources (`add_input_paths`). This method now handles the "placeholder" state (no preset selected) by scanning directories or inspecting archives (ZIP) and creating placeholder `SourceRule`/`AssetRule`/`FileRule` objects to immediately populate the `UnifiedViewModel` with the file structure.
|
||||
* Initiating predictions based on the selected preset mode (Rule-Based or LLM) when presets change or sources are added (`update_preview`).
|
||||
* Starting the processing task (`_on_process_requested`): This slot now filters the `SourceRule` list obtained from the `UnifiedViewModel`, excluding sources where no asset has a `Target Asset` name assigned, before emitting the `start_backend_processing` signal. It also manages enabling/disabling controls.
|
||||
* Managing the background prediction threads (`RuleBasedPredictionHandler` via `QThread`, `LLMPredictionHandler` via `LLMInteractionHandler`).
|
||||
* Implementing slots to handle results from background tasks:
|
||||
* `_on_rule_hierarchy_ready`: Handles results from `RuleBasedPredictionHandler`.
|
||||
* `_on_llm_prediction_ready_from_handler`: Handles results from `LLMInteractionHandler`.
|
||||
* `_on_prediction_error`: Handles errors from both prediction paths.
|
||||
* `_handle_prediction_completion`: Centralized logic to track completion and update UI state after each prediction result or error.
|
||||
* Slots to handle status and state changes from `LLMInteractionHandler`.
|
||||
|
||||
## Threading and Background Tasks
|
||||
|
||||
To keep the UI responsive during intensive operations like asset processing and file preview generation, the GUI utilizes background threads managed by `QThread`.
|
||||
To keep the UI responsive, prediction tasks run in background threads managed by a `QThreadPool`.
|
||||
|
||||
* **`ProcessingHandler` (`gui/processing_handler.py`):** This class is designed to run in a separate `QThread`. It manages the execution of the main asset processing pipeline for multiple assets concurrently using `concurrent.futures.ProcessPoolExecutor`. It submits individual asset processing tasks to the pool and monitors their completion. It uses Qt signals to communicate progress updates, file status changes, and overall processing completion back to the `MainWindow` on the main UI thread. It also handles the execution of optional Blender scripts via subprocess calls after processing. This handler processes and utilizes data structures received from the core processing engine, such as status summaries.
|
||||
* **`PredictionHandler` (`gui/prediction_handler.py`):** This class also runs in a separate `QThread`. It is responsible for generating the detailed file classification previews displayed in the preview table. It calls methods on the `AssetProcessor` (`get_detailed_file_predictions`) to perform the analysis in the background. It uses a `ThreadPoolExecutor` for potentially concurrent prediction tasks. Results are sent back to the `MainWindow` via Qt signals to update the preview table data. This handler works with data structures containing file prediction details.
|
||||
* **`BasePredictionHandler` (`gui/base_prediction_handler.py`):** An abstract `QRunnable` base class defining the common interface and signals (`prediction_signal`, `status_signal`) for prediction tasks.
|
||||
* **`RuleBasedPredictionHandler` (`gui/prediction_handler.py`):** Inherits from `BasePredictionHandler`. Runs as a `QRunnable` in the thread pool when a rule-based preset is selected. Generates the `SourceRule` hierarchy based on preset rules and emits `prediction_signal`.
|
||||
* **`LLMPredictionHandler` (`gui/llm_prediction_handler.py`):** Inherits from `BasePredictionHandler`. Runs as a `QRunnable` in the thread pool when "- LLM Interpretation -" is selected. Interacts with `LLMInteractionHandler`, parses the response, generates the `SourceRule` hierarchy for a *single* input item, and emits `prediction_signal` and `status_signal`.
|
||||
* **`LLMInteractionHandler` (`gui/llm_interaction_handler.py`):** Manages the communication with the LLM service. This handler itself may perform network operations but typically runs synchronously within the `LLMPredictionHandler`'s thread.
|
||||
|
||||
*(Note: The actual processing via `ProcessingEngine` is now handled by `main.ProcessingTask`, which runs in a separate process managed outside the GUI's direct threading model, though the GUI initiates it).*
|
||||
|
||||
## Communication (Signals and Slots)
|
||||
|
||||
Communication between the main UI thread (`MainWindow`) and the background threads (`ProcessingHandler`, `PredictionHandler`) relies heavily on Qt's signals and slots mechanism. This is a thread-safe way for objects in different threads to communicate.
|
||||
Communication between the `MainWindow` (main UI thread) and the background prediction tasks relies on Qt's signals and slots.
|
||||
|
||||
* Background handlers emit signals to indicate events (e.g., progress updated, file status changed, task finished).
|
||||
* The `MainWindow` connects slots (methods) to these signals. When a signal is emitted, the connected slot is invoked on the thread that owns the receiving object (the main UI thread for `MainWindow`), ensuring UI updates happen safely.
|
||||
* Prediction handlers (`RuleBasedPredictionHandler`, `LLMPredictionHandler`) emit signals from the `BasePredictionHandler`:
|
||||
* `prediction_signal(source_id, source_rule_list)`: Indicates prediction for a source is complete.
|
||||
* `status_signal(message)`: Provides status updates (primarily from LLM handler).
|
||||
* The `MainWindow` connects slots to these signals:
|
||||
* `prediction_signal` -> `MainWindow._handle_prediction_completion(source_id, source_rule_list)`
|
||||
* `status_signal` -> `MainWindow._on_status_update(message)` (updates status bar)
|
||||
* Signals from the `UnifiedViewModel` (`dataChanged`, `layoutChanged`) trigger updates in the `QTreeView`.
|
||||
* Signals from the `UnifiedViewModel` (`targetAssetOverrideChanged`) trigger the `AssetRestructureHandler`.
|
||||
|
||||
## Preset Editor
|
||||
## Preset Editor (`gui/preset_editor_widget.py`)
|
||||
|
||||
The GUI includes an integrated preset editor panel. This allows users to interactively create, load, modify, and save preset `.json` files directly within the application. The editor typically uses standard UI widgets to display and edit the key fields of the preset structure.
|
||||
The `PresetEditorWidget` provides a dedicated interface for managing presets. It handles loading, displaying, editing, and saving preset `.json` files.
|
||||
|
||||
## Preview Table
|
||||
* **Refactoring:** This widget has been refactored to expose its main components:
|
||||
* `selector_container`: A `QWidget` containing the preset list (`QListWidget`) and New/Delete buttons. Used statically by `MainWindow`.
|
||||
* `json_editor_container`: A `QWidget` containing the tabbed editor (`QTabWidget`) for preset JSON details and the Save/Save As buttons. Placed in `MainWindow`'s `QStackedWidget`.
|
||||
* **Functionality:** Still manages the logic for populating the preset list, loading/saving presets, handling unsaved changes, and providing the editor UI for preset details.
|
||||
* **Communication:** Emits `preset_selection_changed_signal(mode, preset_name)` when the user selects a preset, the LLM option, or the placeholder. This signal is crucial for `MainWindow` to switch the editor stack and trigger preview updates.
|
||||
|
||||
## LLM Settings Editor (`gui/llm_editor_widget.py`)
|
||||
|
||||
The `PreviewTableModel` receives a list of file prediction dictionaries from the `PredictionHandler` via the `prediction_results_ready` signal. This list contains dictionaries for each file with details such as original path, predicted asset name, status, and other relevant information.
|
||||
This new widget provides a dedicated interface for editing LLM-specific settings stored in `config/llm_settings.json`.
|
||||
|
||||
The `PreviewTableModel` is designed to process and display this file prediction data. Instead of directly displaying the flat list, it processes and transforms the data into a structured list of rows (`self._table_rows`). This transformation involves:
|
||||
* **Purpose:** Allows users to configure the LLM predictor's behavior without directly editing the JSON file.
|
||||
* **Structure:** Uses a `QTabWidget` with two tabs:
|
||||
* **"Prompt Settings":** Contains a `QPlainTextEdit` for the main prompt and a nested `QTabWidget` for managing examples (add/delete/edit JSON in `QTextEdit` widgets).
|
||||
* **"API Settings":** Contains fields (`QLineEdit`, `QDoubleSpinBox`, `QSpinBox`) for endpoint URL, API key, model name, temperature, and timeout.
|
||||
* **Functionality:**
|
||||
* `load_settings()`: Reads `config/llm_settings.json` and populates the UI fields. Handles file not found or JSON errors. Called by `MainWindow` when switching to LLM mode.
|
||||
* `_save_settings()`: Gathers data from the UI, validates example JSON, constructs the settings dictionary, and calls `configuration.save_llm_config()` to write back to the file. Emits `settings_saved` signal on success.
|
||||
* Manages unsaved changes state and enables/disables the "Save LLM Settings" button accordingly.
|
||||
|
||||
1. **Grouping:** Files are grouped based on their `source_asset`.
|
||||
2. **Separation:** Within each asset group, files are separated into `main_files` (Mapped, Model, Error) and `additional_files` (Ignored, Extra, Unrecognised, Unmatched Extra).
|
||||
3. **Structuring Rows:** Rows are created for `self._table_rows` to represent the grouped data. Each row can contain information about a main file and/or an additional file, allowing for the display of additional files in a separate column aligned with the main files of the same asset. Empty rows are created if there are more additional files than main files for an asset to maintain alignment.
|
||||
## Unified Hierarchical View
|
||||
|
||||
The `data()` method of the `PreviewTableModel` then accesses this structured `self._table_rows` list to provide data to the `QTableView` for display. It handles different columns and roles (Display, Tooltip, Foreground, and Background).
|
||||
The core rule editing interface is built around a `QTreeView` managed within the `MainPanelWidget`, using a custom model and delegates.
|
||||
|
||||
* `Qt.ItemDataRole.ForegroundRole`: Used to set the text color of individual cells based on the status of the file they represent. Coloring is applied to cells corresponding to a main file based on the main file's status, and to cells in the "Additional Files" column based on the additional file's status.
|
||||
* `Qt.ItemDataRole.BackgroundRole`: Used to provide alternating background colors based on the index of the asset group the row belongs to in a sorted list of unique assets, improving visual separation between different asset groups.
|
||||
* **`UnifiedViewModel` (`gui/unified_view_model.py`):** Implements `QAbstractItemModel`.
|
||||
* Wraps the `RuleHierarchyModel` to expose the `SourceRule` list (Source -> Asset -> File) to the `QTreeView`.
|
||||
* Provides data for display and flags for editing.
|
||||
* **Handles `setData` requests:** Validates input and updates the underlying `RuleHierarchyModel`. Crucially, it **delegates** complex restructuring (when `target_asset_name_override` changes) to the `AssetRestructureHandler` by emitting the `targetAssetOverrideChanged` signal.
|
||||
* **Row Coloring:** Provides data for `Qt.ForegroundRole` (text color) based on the `item_type` and the colors defined in `config/app_settings.json`. Provides data for `Qt.BackgroundRole` based on calculating a 30% darker shade of the parent asset's background color.
|
||||
* **Caching:** Caches configuration data (`ASSET_TYPE_DEFINITIONS`, `FILE_TYPE_DEFINITIONS`, color maps) in `__init__` for performance.
|
||||
* **`update_rules_for_sources` Method:** Intelligently merges new prediction results or placeholder rules into the existing model data, preserving user overrides where applicable.
|
||||
* *(Note: The previous concept of switching between "simple" and "detailed" display modes has been removed. The model always represents the full detailed structure.)*
|
||||
* **`RuleHierarchyModel` (`gui/rule_hierarchy_model.py`):** A non-Qt model holding the actual list of `SourceRule` objects. Provides methods for accessing and modifying the hierarchy (used by `UnifiedViewModel` and `AssetRestructureHandler`).
|
||||
* **`AssetRestructureHandler` (`gui/asset_restructure_handler.py`):** Contains the logic to modify the `RuleHierarchyModel` when a file's target asset is changed. It listens for the `targetAssetOverrideChanged` signal from the `UnifiedViewModel` and uses methods on the `RuleHierarchyModel` (`moveFileRule`, `createAssetRule`, `removeAssetRule`) to perform the restructuring safely.
|
||||
* **`Delegates` (`gui/delegates.py`):** Custom `QStyledItemDelegate` implementations provide inline editors:
|
||||
* **`ComboBoxDelegate`:** For selecting predefined types (from `Configuration`).
|
||||
* **`LineEditDelegate`:** For free-form text editing.
|
||||
* **`SupplierSearchDelegate`:** For supplier names with auto-completion (using `config/suppliers.json`).
|
||||
|
||||
The `PreviewSortFilterProxyModel` operates on this structured data, implementing a multi-level sort based on source asset, row type (main vs. additional-only), and file paths within those types.
|
||||
|
||||
|
||||
### Preview Table Column Configuration
|
||||
|
||||
The display and behavior of the columns in the `QTableView` are configured in `gui/main_window.py`. The current configuration is as follows:
|
||||
|
||||
* **Column Order (from left to right):**
|
||||
1. Status
|
||||
2. Predicted Asset
|
||||
3. Details
|
||||
4. Original Path
|
||||
5. Additional Files
|
||||
* **Column Resizing:**
|
||||
* Status: Resizes to content.
|
||||
* Predicted Asset: Resizes to content.
|
||||
* Details: Resizes to content.
|
||||
* Original Path: Resizes to content (fixed width behavior).
|
||||
* Additional Files: Stretches to fill available space.
|
||||
|
||||
**Data Flow Diagram:**
|
||||
**Data Flow Diagram (GUI Rule Management - Refactored):**
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[PredictionHandler] -- prediction_results_ready(flat_list) --> B(PreviewTableModel);
|
||||
subgraph PreviewTableModel
|
||||
C[set_data] -- Processes flat_list --> D{Internal Grouping & Transformation};
|
||||
D -- Creates --> E[_table_rows (Structured List)];
|
||||
F[data()] -- Reads from --> E;
|
||||
graph TD
|
||||
subgraph MainWindow [MainWindow Coordinator]
|
||||
direction LR
|
||||
MW_Input[User Input (Drag/Drop)] --> MW(MainWindow);
|
||||
MW -- Owns/Manages --> Splitter(QSplitter);
|
||||
MW -- Owns/Manages --> LLMIH(LLMInteractionHandler);
|
||||
MW -- Owns/Manages --> ARH(AssetRestructureHandler);
|
||||
MW -- Owns/Manages --> VM(UnifiedViewModel);
|
||||
MW -- Owns/Manages --> LCW(LogConsoleWidget);
|
||||
MW -- Initiates --> PredPool{Prediction Threads};
|
||||
MW -- Connects Signals --> VM;
|
||||
MW -- Connects Signals --> ARH;
|
||||
MW -- Connects Signals --> LLMIH;
|
||||
MW -- Connects Signals --> PEW(PresetEditorWidget);
|
||||
MW -- Connects Signals --> LLMEDW(LLMEditorWidget);
|
||||
end
|
||||
B -- Provides data via data() --> G(QTableView via Proxy);
|
||||
|
||||
style B fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style C fill:#ccf,stroke:#333,stroke-width:1px
|
||||
style D fill:#lightgrey,stroke:#333,stroke-width:1px
|
||||
style E fill:#ccf,stroke:#333,stroke-width:1px
|
||||
style F fill:#ccf,stroke:#333,stroke-width:1px
|
||||
subgraph LeftPane [Left Pane Widgets]
|
||||
direction TB
|
||||
Splitter -- Adds Widget --> LPW(Left Pane Container);
|
||||
LPW -- Contains --> PEW_Sel(PresetEditorWidget - Selector);
|
||||
LPW -- Contains --> Stack(QStackedWidget);
|
||||
Stack -- Contains --> PEW_Edit(PresetEditorWidget - JSON Editor);
|
||||
Stack -- Contains --> LLMEDW;
|
||||
end
|
||||
|
||||
subgraph RightPane [Right Pane Widgets]
|
||||
direction TB
|
||||
Splitter -- Adds Widget --> MPW(MainPanelWidget);
|
||||
MPW -- Contains --> TV(QTreeView - Rule View);
|
||||
MPW_UI[UI Controls (Process Btn, etc)];
|
||||
MPW_UI --> MPW;
|
||||
end
|
||||
|
||||
subgraph Prediction [Background Prediction]
|
||||
direction TB
|
||||
PredPool -- Runs --> RBP(RuleBasedPredictionHandler);
|
||||
PredPool -- Runs --> LLMP(LLMPredictionHandler);
|
||||
LLMIH -- Manages/Starts --> LLMP;
|
||||
RBP -- prediction_ready/error/status --> MW;
|
||||
LLMIH -- llm_prediction_ready/error/status --> MW;
|
||||
end
|
||||
|
||||
subgraph ModelView [Model/View Components]
|
||||
direction TB
|
||||
TV -- Sets Model --> VM;
|
||||
TV -- Displays Data From --> VM;
|
||||
TV -- Uses Delegates --> Del(Delegates);
|
||||
UserEdit[User Edits Rules] --> TV;
|
||||
TV -- setData --> VM;
|
||||
VM -- Wraps --> RHM(RuleHierarchyModel);
|
||||
VM -- dataChanged/layoutChanged --> TV;
|
||||
VM -- targetAssetOverrideChanged --> ARH;
|
||||
ARH -- Modifies --> RHM;
|
||||
Del -- Get/Set Data --> VM;
|
||||
end
|
||||
|
||||
%% MainWindow Interactions
|
||||
MW_Input -- Triggers --> MW;
|
||||
PEW -- preset_selection_changed_signal --> MW;
|
||||
LLMEDW -- settings_saved --> MW;
|
||||
MPW -- process_requested/etc --> MW;
|
||||
MW -- _on_preset_selection_changed --> Stack;
|
||||
MW -- _on_preset_selection_changed --> LLMEDW;
|
||||
MW -- _handle_prediction_completion --> VM;
|
||||
MW -- Triggers Processing --> ProcTask(main.ProcessingTask);
|
||||
|
||||
%% Connections between subgraphs
|
||||
PEW --> LPW; %% PresetEditorWidget parts are in Left Pane
|
||||
LLMEDW --> Stack; %% LLMEditorWidget is in Stack
|
||||
MPW --> Splitter; %% MainPanelWidget is in Right Pane
|
||||
VM --> MW;
|
||||
ARH --> MW;
|
||||
LLMIH --> MW;
|
||||
LCW --> MW;
|
||||
```
|
||||
|
||||
### Application Styling
|
||||
## Application Styling
|
||||
|
||||
The application style is explicitly set to 'Fusion' in `gui/main_window.py` to provide a more consistent look and feel across different operating systems, particularly to address styling inconsistencies observed on Windows 11. A custom `QPalette` is also applied to the application to adjust default colors within the 'Fusion' style, specifically to change the background color of list-like widgets and potentially other elements from a default dark blue to a more neutral grey.
|
||||
The application style is explicitly set to 'Fusion' in `gui/main_window.py`. A custom `QPalette` adjusts default colors.
|
||||
|
||||
## Logging
|
||||
## Logging (`gui/log_console_widget.py`)
|
||||
|
||||
A custom `QtLogHandler` is used to redirect log messages from the standard Python `logging` module to a text area or console widget within the GUI, allowing users to see detailed application output and errors.
|
||||
The `LogConsoleWidget` displays logs captured by a custom `QtLogHandler` from Python's `logging` module.
|
||||
|
||||
## Cancellation
|
||||
|
||||
The GUI provides a "Cancel" button to stop ongoing processing. The `ProcessingHandler` implements logic to handle cancellation requests. This typically involves setting an internal flag and attempting to shut down the `ProcessPoolExecutor`. However, it's important to note that this does not immediately terminate worker processes that are already executing; it primarily prevents new tasks from starting and stops processing results from completed futures once the cancellation flag is checked.
|
||||
The GUI provides a "Cancel" button. Cancellation logic for the actual processing is now likely handled within the `main.ProcessingTask` or the code that manages it, as the `ProcessingHandler` has been removed. The GUI button would signal this external task manager.
|
||||
|
||||
**Note on Data Passing:** As mentioned in the Architecture documentation, major changes to the data passing mechanisms between the GUI, Main (CLI orchestration), and `AssetProcessor` modules are currently being planned. The descriptions of how data is handled and passed within the GUI and its interactions with background handlers reflect the current state and will require review and updates once the plan for these changes is finalized.
|
||||
## Application Preferences Editor (`gui/config_editor_dialog.py`)
|
||||
|
||||
A dedicated dialog for editing user-overridable application settings. It loads base settings from `config/app_settings.json` and saves user overrides to `config/user_settings.json`.
|
||||
|
||||
* **Functionality:** Provides a tabbed interface to edit various application settings, including general paths, output/naming patterns, image processing options (like resolutions and compression), and map merging rules. It no longer includes editors for Asset Type or File Type Definitions.
|
||||
* **Integration:** Launched by `MainWindow` via the "Edit" -> "Preferences..." menu.
|
||||
* **Persistence:** Saves changes to `config/user_settings.json`. Changes require an application restart to take effect in processing logic.
|
||||
|
||||
The refactored GUI separates concerns into distinct widgets and handlers, coordinated by the `MainWindow`. Background tasks use `QThreadPool` and `QRunnable`. The `UnifiedViewModel` focuses on data presentation and simple edits, delegating complex restructuring to the `AssetRestructureHandler`.
|
||||
|
||||
## Definitions Editor (`gui/definitions_editor_dialog.py`)
|
||||
|
||||
A new dedicated dialog for managing core application definitions that are separate from general user preferences.
|
||||
|
||||
* **Purpose:** Provides a structured UI for editing Asset Type Definitions, File Type Definitions, and Supplier Settings.
|
||||
* **Structure:** Uses a `QTabWidget` with three tabs:
|
||||
* **Asset Type Definitions:** Manages definitions from `config/asset_type_definitions.json`. Presents a list of asset types and allows editing their description, color, and examples.
|
||||
* **File Type Definitions:** Manages definitions from `config/file_type_definitions.json`. Presents a list of file types and allows editing their description, color, examples, standard type, bit depth rule, grayscale status, and keybind.
|
||||
* **Supplier Settings:** Manages settings from `config/suppliers.json`. Presents a list of suppliers and allows editing supplier-specific settings (e.g., Normal Map Type).
|
||||
* **Integration:** Launched by `MainWindow` via the "Edit" -> "Edit Definitions..." menu.
|
||||
* **Persistence:** Saves changes directly to the respective configuration files (`config/asset_type_definitions.json`, `config/file_type_definitions.json`, `config/suppliers.json`). Some changes may require an application restart.
|
||||
@ -4,30 +4,41 @@ This document provides technical details about the implementation of the Directo
|
||||
|
||||
## Overview
|
||||
|
||||
The `monitor.py` script provides an automated way to process assets by monitoring a specified input directory for new ZIP files. It is built using the `watchdog` library.
|
||||
The `monitor.py` script provides an automated way to process assets by monitoring a specified input directory for new archive files. It has been refactored to use a `ThreadPoolExecutor` for asynchronous processing.
|
||||
|
||||
## Key Components
|
||||
|
||||
* **`watchdog` Library:** The script relies on the `watchdog` library for monitoring file system events. Specifically, it uses a `PollingObserver` to watch the `INPUT_DIR` for changes.
|
||||
* **`ZipHandler` Class:** This is a custom event handler class defined within `monitor.py`. It inherits from a `watchdog` event handler class (likely `FileSystemEventHandler` or similar, though not explicitly stated in the source text, it's the standard pattern). Its primary method of interest is the one that handles file creation events (`on_created`).
|
||||
* **`main.run_processing`:** The monitor script triggers the main asset processing logic by calling the `run_processing` function from the `main.py` module.
|
||||
* **`watchdog` Library:** Used for monitoring file system events (specifically file creation) in the `INPUT_DIR`. A `PollingObserver` is typically used.
|
||||
* **`concurrent.futures.ThreadPoolExecutor`:** Manages a pool of worker threads to process detected archives concurrently. The number of workers can often be configured (e.g., via `NUM_WORKERS` environment variable).
|
||||
* **`_process_archive_task` Function:** The core function executed by the thread pool for each detected archive. It encapsulates the entire processing workflow for a single archive.
|
||||
* **`utils.prediction_utils.generate_source_rule_from_archive`:** A utility function called by `_process_archive_task` to perform rule-based prediction directly on the archive file and generate the necessary `SourceRule` object.
|
||||
* **`utils.workspace_utils.prepare_processing_workspace`:** A utility function called by `_process_archive_task` to create a temporary workspace and extract the archive contents into it.
|
||||
* **`ProcessingEngine` (`processing_engine.py`):** The core engine instantiated and run within `_process_archive_task` to perform the actual asset processing based on the generated `SourceRule`.
|
||||
* **`Configuration` (`configuration.py`):** Loaded within `_process_archive_task` based on the preset derived from the archive filename.
|
||||
|
||||
## Functionality Details
|
||||
## Functionality Details (Asynchronous Workflow)
|
||||
|
||||
1. **Watching:** A `PollingObserver` is set up to monitor the directory specified by the `INPUT_DIR` environment variable. Polling is used, checking for changes at a frequency defined by `POLL_INTERVAL`.
|
||||
2. **Event Handling:** The `ZipHandler` is attached to the observer. When a file is created in the monitored directory, the `on_created` method of the `ZipHandler` is triggered.
|
||||
3. **ZIP File Detection:** The `on_created` method checks if the newly created file is a `.zip` file.
|
||||
4. **Filename Parsing:** If it's a ZIP file, the script expects the filename to follow a specific format: `[preset]_filename.zip`. It uses a regular expression (`PRESET_FILENAME_REGEX`, likely defined in `config.py` or similar) to extract the `[preset]` part from the filename.
|
||||
5. **Preset Validation:** It validates whether the extracted `preset` name corresponds to an existing preset JSON file in the `Presets/` directory.
|
||||
6. **Triggering Processing:** If the preset is valid, the `monitor.py` script calls `main.run_processing`, passing the path to the detected ZIP file and the extracted preset name. This initiates the main asset processing pipeline for that single asset. A `PROCESS_DELAY` can be configured to wait before triggering processing, potentially allowing large files to finish copying.
|
||||
7. **Source ZIP Management:** After the processing initiated by `main.run_processing` completes, the original source `.zip` file is moved to either the `PROCESSED_DIR` (if processing was successful or skipped) or the `ERROR_DIR` (if processing failed or the preset was invalid).
|
||||
1. **Watching:** A `watchdog` observer monitors the `INPUT_DIR` for file creation events.
|
||||
2. **Event Handling:** When a file is created, an event handler (e.g., `on_created` method) is triggered.
|
||||
3. **Archive Detection:** The handler checks if the new file is a supported archive type (e.g., `.zip`, `.rar`, `.7z`).
|
||||
4. **Filename Parsing:** If it's a supported archive, the script attempts to parse the filename to extract the intended preset name (e.g., using a regex like `[preset]_filename.ext`).
|
||||
5. **Preset Validation:** The extracted preset name is validated against existing preset files (`Presets/*.json`).
|
||||
6. **Task Submission:** If the preset is valid, the path to the archive file and the validated preset name are submitted as a task to the `ThreadPoolExecutor`, which will eventually run the `_process_archive_task` function with these arguments in a worker thread.
|
||||
7. **`_process_archive_task` Execution (Worker Thread):**
|
||||
* **Load Configuration:** Loads the `Configuration` object using the provided preset name.
|
||||
* **Generate SourceRule:** Calls `utils.prediction_utils.generate_source_rule_from_archive`, passing the archive path and `Configuration`. This utility handles temporary extraction (if needed internally) and rule-based prediction, returning the `SourceRule`.
|
||||
* **Prepare Workspace:** Calls `utils.workspace_utils.prepare_processing_workspace`, passing the archive path. This creates a unique temporary directory and extracts the archive contents. It returns the path to the prepared workspace. This step should ideally be wrapped in a `try...finally` block to ensure cleanup.
|
||||
* **Instantiate Engine:** Creates an instance of the `ProcessingEngine`, passing the loaded `Configuration` and the prepared workspace path.
|
||||
* **Run Processing:** Calls the `ProcessingEngine.process()` method, passing the generated `SourceRule`.
|
||||
* **Handle Results:** Based on the success or failure of the processing, moves the original source archive file from `INPUT_DIR` to either `PROCESSED_DIR` or `ERROR_DIR`.
|
||||
* **Cleanup Workspace:** Ensures the temporary workspace directory created by `prepare_processing_workspace` is removed (e.g., in the `finally` block).
|
||||
|
||||
## Configuration
|
||||
|
||||
The monitor's behavior is primarily controlled by environment variables, which are read by the `monitor.py` script. These include `INPUT_DIR`, `OUTPUT_DIR`, `PROCESSED_DIR`, `ERROR_DIR`, `LOG_LEVEL`, `POLL_INTERVAL`, and `NUM_WORKERS`.
|
||||
The monitor's behavior is controlled by environment variables or configuration settings, likely including `INPUT_DIR`, `OUTPUT_DIR`, `PROCESSED_DIR`, `ERROR_DIR`, `LOG_LEVEL`, `POLL_INTERVAL`, and potentially `NUM_WORKERS` to control the size of the `ThreadPoolExecutor`.
|
||||
|
||||
## Limitations
|
||||
|
||||
* The current implementation of the directory monitor does *not* support triggering the optional Blender script execution after processing. This post-processing step is only available when running the tool via the CLI or GUI.
|
||||
* The monitor likely still does *not* support triggering optional Blender script execution post-processing, as this integration point was complex and potentially removed or not yet reimplemented in the refactored workflow.
|
||||
|
||||
Understanding the interaction between `watchdog`, the `ZipHandler`, and the call to `main.run_processing` is key to debugging or modifying the directory monitoring functionality.
|
||||
Understanding the asynchronous nature, the role of the `ThreadPoolExecutor`, the `_process_archive_task` function, and the reliance on utility modules (`prediction_utils`, `workspace_utils`) is key to debugging or modifying the directory monitoring functionality.
|
||||
@ -20,10 +20,10 @@ This document outlines the coding conventions and general practices followed wit
|
||||
* Use Qt's signals and slots mechanism for communication between objects, especially across threads.
|
||||
* Run long-running or blocking tasks in separate `QThread`s to keep the main UI thread responsive.
|
||||
* Perform UI updates only from the main UI thread.
|
||||
* **Configuration:** Core settings are managed in `config.py` (Python module). Supplier-specific rules are managed in JSON files (`Presets/`). The `Configuration` class handles loading and merging these.
|
||||
* **Configuration:** Core application settings are defined in `config/app_settings.json`. Supplier-specific rules are managed in JSON files within the `Presets/` directory. The `Configuration` class (`configuration.py`) is responsible for loading `app_settings.json` and merging it with the selected preset file.
|
||||
* **File Paths:** Use `pathlib.Path` objects for handling file system paths. Avoid using string manipulation for path joining or parsing.
|
||||
* **Docstrings:** Write clear and concise docstrings for modules, classes, methods, and functions, explaining their purpose, arguments, and return values.
|
||||
* **Comments:** Use comments to explain complex logic or non-obvious parts of the code.
|
||||
* **Comments:** Use comments to explain complex logic or non-obvious parts of the code. Avoid obsolete comments (e.g., commented-out old code) and redundant comments (e.g., comments stating the obvious, like `# Import module` or `# Initialize variable`). The goal is to maintain clarity while minimizing unnecessary token usage for LLM tools.
|
||||
* **Imports:** Organize imports at the top of the file, grouped by standard library, third-party libraries, and local modules.
|
||||
* **Naming:**
|
||||
* Use `snake_case` for function and variable names.
|
||||
@ -31,4 +31,51 @@ This document outlines the coding conventions and general practices followed wit
|
||||
* Use `UPPER_CASE` for constants.
|
||||
* Use a leading underscore (`_`) for internal or "protected" methods/attributes.
|
||||
|
||||
## Terminology and Data Standards
|
||||
|
||||
To ensure consistency and clarity across the codebase, particularly concerning asset and file classifications, the following standards must be adhered to. These primarily revolve around definitions stored in `config/app_settings.json`.
|
||||
|
||||
### `FILE_TYPE_DEFINITIONS`
|
||||
|
||||
`FILE_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all file type identifiers used within the application.
|
||||
|
||||
* **`FileRule.item_type` and `FileRule.item_type_override`**: When defining or interpreting `SourceRule` objects (and their constituent `FileRule` instances), the `item_type` and `item_type_override` attributes **must** always use a key directly from `FILE_TYPE_DEFINITIONS`.
|
||||
* Example: `file_rule.item_type = "MAP_COL"` (for a color map) or `file_rule.item_type = "MODEL_FBX"` (for an FBX model).
|
||||
|
||||
* **`standard_type` Property**: Each entry in `FILE_TYPE_DEFINITIONS` includes a `standard_type` property. This provides a common, often abbreviated, alias for the file type.
|
||||
* Example: `FILE_TYPE_DEFINITIONS["MAP_COL"]["standard_type"]` might be `"COL"`.
|
||||
* Example: `FILE_TYPE_DEFINITIONS["MAP_NORM_GL"]["standard_type"]` might be `"NRM"`.
|
||||
|
||||
* **Removal of `STANDARD_MAP_TYPES`**: The global constant `STANDARD_MAP_TYPES` (previously in `config.py`) has been **removed**. Standard map type aliases (e.g., "COL", "NRM", "RGH") are now derived dynamically from the `standard_type` property of the relevant entry in `FILE_TYPE_DEFINITIONS`.
|
||||
|
||||
* **`map_type` Usage**:
|
||||
* **Filename Tokens**: When used as a token in output filename patterns (e.g., `[maptype]`), `map_type` is typically derived from the `standard_type` of the file's effective `item_type`. It may also include a variant suffix if applicable (e.g., "COL", "COL_var01").
|
||||
* **General Classification**: For precise classification within code logic or rules, developers should refer to the full `FILE_TYPE_DEFINITIONS` key (e.g., `"MAP_COL"`, `"MAP_METAL"`). The `standard_type` can be used for broader categorization or when a common alias is needed.
|
||||
|
||||
* **`Map_type_Mapping.target_type` in Presets**: Within preset files (e.g., `Presets/Poliigon.json`), the `map_type_mapping` rules found inside a `FileRule`'s `map_processing_options` now use keys from `FILE_TYPE_DEFINITIONS` for the `target_type` field.
|
||||
* Example:
|
||||
```json
|
||||
// Inside a FileRule in a preset
|
||||
"map_processing_options": {
|
||||
"map_type_mapping": {
|
||||
"source_type_pattern": ".*ambient occlusion.*",
|
||||
"target_type": "MAP_AO", // Uses FILE_TYPE_DEFINITIONS key
|
||||
"source_channels": "RGB"
|
||||
}
|
||||
}
|
||||
```
|
||||
This replaces old aliases like `"AO"` or `"OCC"`.
|
||||
|
||||
* **`is_grayscale` Property**: `FILE_TYPE_DEFINITIONS` entries can now include an `is_grayscale` boolean property. This flag indicates whether the file type is inherently grayscale (e.g., a roughness map). It can be used by the processing engine to inform decisions about channel handling, compression, or specific image operations.
|
||||
* Example: `FILE_TYPE_DEFINITIONS["MAP_RGH"]["is_grayscale"]` might be `true`.
|
||||
|
||||
### `ASSET_TYPE_DEFINITIONS`
|
||||
|
||||
Similarly, `ASSET_TYPE_DEFINITIONS` in `config/app_settings.json` is the **single source of truth** for all asset type identifiers.
|
||||
|
||||
* **`AssetRule.asset_type`, `AssetRule.asset_type_override`, and `AssetRule.asset_category`**: When defining or interpreting `SourceRule` objects (and their constituent `AssetRule` instances), the `asset_type`, `asset_type_override`, and `asset_category` attributes **must** always use a key directly from `ASSET_TYPE_DEFINITIONS`.
|
||||
* Example: `asset_rule.asset_type = "SURFACE_3D"` or `asset_rule.asset_category = "FABRIC"`.
|
||||
|
||||
Adherence to these definitions ensures that terminology remains consistent throughout the application, from configuration to core logic and output.
|
||||
|
||||
Adhering to these conventions will make the codebase more consistent, easier to understand, and more maintainable for all contributors.
|
||||
123
Documentation/02_Developer_Guide/12_LLM_Predictor_Integration.md
Normal file
123
Documentation/02_Developer_Guide/12_LLM_Predictor_Integration.md
Normal file
@ -0,0 +1,123 @@
|
||||
# LLM Predictor Integration
|
||||
|
||||
## Overview
|
||||
|
||||
The LLM Predictor feature provides an alternative method for classifying asset textures using a Large Language Model (LLM). This allows for more flexible and potentially more accurate classification compared to traditional rule-based methods, especially for diverse or complex asset names.
|
||||
|
||||
## Configuration
|
||||
|
||||
The LLM Predictor is configured via settings in the dedicated `config/llm_settings.json` file. These settings control the behavior of the LLM interaction:
|
||||
|
||||
- `llm_predictor_prompt`: The template for the prompt sent to the LLM. This prompt should guide the LLM to classify the asset based on its name and potentially other context. It can include placeholders that will be replaced with actual data during processing.
|
||||
- `llm_endpoint_url`: The URL of the LLM API endpoint.
|
||||
- `llm_api_key`: The API key required for authentication with the LLM endpoint.
|
||||
- `llm_model_name`: The name of the specific LLM model to be used for prediction.
|
||||
- `llm_temperature`: Controls the randomness of the LLM's output. A lower temperature results in more deterministic output, while a higher temperature increases creativity.
|
||||
- `llm_request_timeout`: The maximum time (in seconds) to wait for a response from the LLM API.
|
||||
- `llm_predictor_examples`: A list of example input/output pairs to include in the prompt for few-shot learning, helping the LLM understand the desired output format and classification logic.
|
||||
|
||||
**Editing:** These settings can be edited directly through the GUI using the **`LLMEditorWidget`** (`gui/llm_editor_widget.py`), which provides a user-friendly interface for modifying the prompt, examples, and API parameters. Changes are saved back to `config/llm_settings.json` via the `configuration.save_llm_config()` function.
|
||||
|
||||
**Loading:** The `LLMInteractionHandler` now loads these settings directly from `config/llm_settings.json` and relevant parts of `config/app_settings.json` when it needs to start an `LLMPredictionHandler` task. It no longer relies on the main `Configuration` class for LLM-specific settings. The prompt structure remains crucial for effective classification. Placeholders within the prompt template (e.g., `{FILE_LIST}`) are dynamically replaced with relevant data before the request is sent.
|
||||
|
||||
## Expected LLM Output Format (Refactored)
|
||||
|
||||
The LLM is now expected to return a JSON object containing two distinct parts. This structure helps the LLM maintain context across multiple files belonging to the same conceptual asset and allows for a more robust grouping mechanism.
|
||||
|
||||
**Rationale:** The previous implicit format made it difficult for the LLM to consistently group related files (e.g., different texture maps for the same material) under a single asset, especially in complex archives. The new two-part structure explicitly separates file-level analysis from asset-level classification, improving accuracy and consistency.
|
||||
|
||||
**Structure:**
|
||||
|
||||
```json
|
||||
{
|
||||
"individual_file_analysis": [
|
||||
{
|
||||
"relative_file_path": "Textures/Wood_Floor_01/Wood_Floor_01_BaseColor.png",
|
||||
"classified_file_type": "BaseColor",
|
||||
"proposed_asset_group_name": "Wood_Floor_01"
|
||||
},
|
||||
{
|
||||
"relative_file_path": "Textures/Wood_Floor_01/Wood_Floor_01_Roughness.png",
|
||||
"classified_file_type": "Roughness",
|
||||
"proposed_asset_group_name": "Wood_Floor_01"
|
||||
},
|
||||
{
|
||||
"relative_file_path": "Textures/Metal_Plate_03/Metal_Plate_03_Metallic.jpg",
|
||||
"classified_file_type": "Metallic",
|
||||
"proposed_asset_group_name": "Metal_Plate_03"
|
||||
}
|
||||
],
|
||||
"asset_group_classifications": {
|
||||
"Wood_Floor_01": "PBR Material",
|
||||
"Metal_Plate_03": "PBR Material"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- **`individual_file_analysis`**: A list where each object represents a single file within the source.
|
||||
- `relative_file_path`: The path of the file relative to the source root.
|
||||
- `classified_file_type`: The LLM's prediction for the *type* of this specific file (e.g., "BaseColor", "Normal", "Model"). This corresponds to the `item_type` in the `FileRule`.
|
||||
- `proposed_asset_group_name`: A name suggested by the LLM to group this file with others belonging to the same conceptual asset. This is used internally by the parser.
|
||||
- **`asset_group_classifications`**: A dictionary mapping the `proposed_asset_group_name` values from the list above to a final `asset_type` (e.g., "PBR Material", "HDR Environment").
|
||||
|
||||
## `LLMInteractionHandler` (Refactored)
|
||||
|
||||
The `gui/llm_interaction_handler.py` module contains the `LLMInteractionHandler` class, which now acts as the central manager for LLM prediction tasks.
|
||||
|
||||
Key Responsibilities & Methods:
|
||||
|
||||
- **Queue Management:** Maintains a queue (`llm_processing_queue`) of pending prediction requests (input path, file list). Handles adding single (`queue_llm_request`) or batch (`queue_llm_requests_batch`) requests.
|
||||
- **State Management:** Tracks whether an LLM task is currently running (`_is_processing`) and emits `llm_processing_state_changed(bool)` to update the GUI (e.g., disable preset editor). Includes `force_reset_state()` for recovery.
|
||||
- **Task Orchestration:** Processes the queue sequentially (`_process_next_llm_item`). For each item:
|
||||
* Loads required settings directly from `config/llm_settings.json` and `config/app_settings.json`.
|
||||
* Instantiates an `LLMPredictionHandler` in a new `QThread`.
|
||||
* Passes the loaded settings dictionary to the `LLMPredictionHandler`.
|
||||
* Connects signals from the handler (`prediction_ready`, `prediction_error`, `status_update`) to internal slots (`_handle_llm_result`, `_handle_llm_error`) or directly re-emits them (`llm_status_update`).
|
||||
* Starts the thread.
|
||||
- **Result/Error Handling:** Internal slots (`_handle_llm_result`, `_handle_llm_error`) receive results/errors from the `LLMPredictionHandler`, remove the completed/failed item from the queue, emit the corresponding public signal (`llm_prediction_ready`, `llm_prediction_error`), and trigger processing of the next queue item.
|
||||
- **Communication:** Emits signals to `MainWindow`:
|
||||
* `llm_prediction_ready(input_path, source_rule_list)`
|
||||
* `llm_prediction_error(input_path, error_message)`
|
||||
* `llm_status_update(status_message)`
|
||||
* `llm_processing_state_changed(is_processing)`
|
||||
|
||||
## `LLMPredictionHandler` (Refactored)
|
||||
|
||||
The `gui/llm_prediction_handler.py` module contains the `LLMPredictionHandler` class (inheriting from `BasePredictionHandler`), which performs the actual LLM prediction for a *single* input source. It runs in a background thread managed by the `LLMInteractionHandler`.
|
||||
|
||||
Key Responsibilities & Methods:
|
||||
|
||||
- **Initialization**: Takes the source identifier, file list, and a **`settings` dictionary** (passed from `LLMInteractionHandler`) containing all necessary configuration (LLM endpoint, prompt, examples, API details, type definitions, etc.).
|
||||
- **`_perform_prediction()`**: Implements the core prediction logic:
|
||||
* **Prompt Preparation (`_prepare_prompt`)**: Uses the passed `settings` dictionary to access the prompt template, type definitions, and examples to build the final prompt string.
|
||||
* **API Call (`_call_llm`)**: Uses the passed `settings` dictionary to get the endpoint URL, API key, model name, temperature, and timeout to make the API request.
|
||||
* **Parsing (`_parse_llm_response`)**: Parses the LLM's JSON response (using type definitions from the `settings` dictionary for validation) and constructs the `SourceRule` hierarchy based on the two-part format (`individual_file_analysis`, `asset_group_classifications`). Includes sanitization logic for comments and markdown fences.
|
||||
- **Signals (Inherited):** Emits `prediction_ready(input_path, source_rule_list)` or `prediction_error(input_path, error_message)` upon completion or failure, which are connected to the `LLMInteractionHandler`. Also emits `status_update(message)`.
|
||||
|
||||
## GUI Integration
|
||||
|
||||
- The LLM predictor mode is selected via the preset dropdown in `PresetEditorWidget`.
|
||||
- Selecting "LLM Interpretation" triggers `MainWindow._on_preset_selection_changed`, which switches the editor view to the `LLMEditorWidget` and calls `update_preview`.
|
||||
- `MainWindow.update_preview` (or `add_input_paths`) delegates the LLM prediction request(s) to the `LLMInteractionHandler`'s queue.
|
||||
- `LLMInteractionHandler` manages the background tasks and signals results/errors/status back to `MainWindow`.
|
||||
- `MainWindow` slots (`_on_llm_prediction_ready_from_handler`, `_on_prediction_error`, `show_status_message`, `_on_llm_processing_state_changed`) handle these signals to update the `UnifiedViewModel` and the UI state (status bar, progress, button enablement).
|
||||
- The `LLMEditorWidget` allows users to modify settings, saving them via `configuration.save_llm_config()`. `MainWindow` listens for the `settings_saved` signal to provide user feedback.
|
||||
|
||||
## Model Integration (Refactored)
|
||||
|
||||
The `gui/unified_view_model.py` module's `update_rules_for_sources` method still incorporates the results.
|
||||
|
||||
- When the `prediction_signal` is received from `LLMPredictionHandler`, the accompanying `SourceRule` object (which has already been constructed based on the new two-part JSON parsing logic) is passed to `update_rules_for_sources`.
|
||||
- This method then merges the new `SourceRule` hierarchy into the existing model data, preserving user overrides where applicable. The internal structure of the received `SourceRule` now directly reflects the groupings and classifications determined by the LLM and the new parser.
|
||||
|
||||
## Error Handling (Updated)
|
||||
|
||||
Error handling is distributed:
|
||||
|
||||
- **Configuration Loading:** `LLMInteractionHandler` handles errors loading `llm_settings.json` or `app_settings.json` before starting a task.
|
||||
- **LLM API Errors:** Handled within `LLMPredictionHandler._call_llm` (e.g., `requests.exceptions.RequestException`, `HTTPError`) and propagated via the `prediction_error` signal.
|
||||
- **Sanitization/Parsing Errors:** `LLMPredictionHandler._parse_llm_response` catches errors during comment/markdown removal and `json.loads()`.
|
||||
- **Structure/Validation Errors:** `LLMPredictionHandler._parse_llm_response` includes explicit checks for the required two-part JSON structure and data consistency.
|
||||
- **Task Management Errors:** `LLMInteractionHandler` handles errors during thread setup/start.
|
||||
|
||||
All errors ultimately result in the `llm_prediction_error` signal being emitted by `LLMInteractionHandler`, allowing `MainWindow` to inform the user via the status bar and handle the completion state.
|
||||
@ -1,55 +0,0 @@
|
||||
# Developer Guide: LLM Integration Progress
|
||||
|
||||
This document summarizes the goals, approach, and current progress on integrating Large Language Model (LLM) capabilities into the Asset Processor Tool for handling irregularly named asset inputs.
|
||||
|
||||
## 1. Initial Goal
|
||||
|
||||
The primary goal is to enhance the Asset Processor Tool's ability to process asset sources with irregular or non-standard naming conventions that cannot be reliably handled by the existing regex and keyword-based preset system. This involves leveraging an LLM to interpret lists of filenames and determine asset metadata and file classifications.
|
||||
|
||||
## 2. Agreed Approach
|
||||
|
||||
After initial discussion and exploring several options, the agreed approach for developing this feature is as follows:
|
||||
|
||||
* **Dedicated LLM Preset:** The LLM classification logic will be triggered by selecting a specific preset type (or flag) in the main tool, indicating that standard rule-based processing should be bypassed in favor of the LLM.
|
||||
* **Standalone Prototype:** The core LLM interaction and classification logic is being developed as a standalone Python prototype within the `llm_prototype/` directory. This allows for focused development, testing, and refinement in isolation before integration into the main application.
|
||||
* **Configurable LLM Endpoint:** The prototype is designed to allow users to configure the LLM API endpoint, supporting various providers including local LLMs (e.g., via LM Studio) and commercial APIs. API keys are handled via environment variables for security.
|
||||
* **Multi-Asset Handling:** The prototype is being built to handle input sources that contain multiple distinct assets within a single directory or archive. The LLM is expected to identify these separate assets and return a JSON **list**, where each item in the list represents one asset.
|
||||
* **Chain of Thought (CoT) Prompting:** To improve the LLM's ability to handle the complex task of identifying multiple assets and classifying files, the prompt includes instructions for the LLM to output its reasoning process within <thinking> tags before generating the final JSON list.
|
||||
* **Unified Asset Category:** The asset classification uses a single `asset_category` field with defined valid values: `Model`, `Surface`, `Decal`, `ATLAS`, `Imperfection`.
|
||||
* **Robust JSON Extraction & Validation:** The prototype includes logic to extract the JSON list from the LLM's response (handling potential extra text) and validate its structure and content against expected schemas and values.
|
||||
|
||||
## 3. Prototype Development Progress
|
||||
|
||||
The initial structure for the standalone prototype has been created in the `llm_prototype/` directory:
|
||||
|
||||
* `llm_prototype/PLAN.md`: This document outlines the detailed plan.
|
||||
* `llm_prototype/config_llm.py`: Configuration file for LLM settings, expected values, and placeholders.
|
||||
* `llm_prototype/llm_classifier.py`: Main script containing the core logic (loading config/input/prompt, formatting prompt, calling LLM API, extracting/validating JSON).
|
||||
* `llm_prototype/requirements_llm.txt`: Lists the `requests` library dependency.
|
||||
* `llm_prototype/prompt_template.txt`: Contains the Chain of Thought prompt template with placeholders and few-shot examples.
|
||||
* `llm_prototype/README.md`: Provides setup and running instructions for the prototype.
|
||||
* `llm_prototype/test_inputs/`: Contains example input JSON files (`dinesen_example.json`, `imperfections_example.json`) representing file lists from asset sources.
|
||||
|
||||
Code has been added to `llm_classifier.py` for loading inputs/config/prompt, formatting the prompt, calling the API, and extracting/validating the JSON response. The JSON extraction logic has been made more robust to handle potential variations in LLM output format.
|
||||
|
||||
## 4. Current Status and Challenges
|
||||
|
||||
Initial testing of the prototype revealed the following:
|
||||
|
||||
* Successful communication with the configured LLM API endpoint.
|
||||
* The LLM is attempting to follow the Chain of Thought structure and generate the list-based JSON output.
|
||||
* **Challenge:** The LLM is currently failing to consistently produce complete and valid JSON output, leading to JSON decoding errors in the prototype script.
|
||||
* **Challenge:** The LLM is not strictly adhering to the specified classification values (e.g., returning "Map" instead of "PBRMap"), despite the prompt explicitly listing the allowed values and including few-shot examples.
|
||||
|
||||
To address these challenges, few-shot examples demonstrating the expected JSON structure and exact classification values were added to the `prompt_template.txt`. The JSON extraction logic in `llm_classifier.py` was also updated to be more resilient.
|
||||
|
||||
## 5. Next Steps
|
||||
|
||||
The immediate next steps are focused on debugging and improving the LLM's output reliability:
|
||||
|
||||
1. Continue testing the prototype with the updated `prompt_template.txt` (including examples) using the example input files.
|
||||
2. Analyze the terminal output to determine if the few-shot examples and improved extraction logic have resolved the JSON completeness and classification value issues.
|
||||
3. Based on the results, iterate on the prompt template (e.g., further emphasizing strict adherence to output format and values) and/or the JSON extraction/validation logic in `llm_classifier.py` as needed.
|
||||
4. Repeat testing and iteration until the prototype reliably produces valid JSON output with correct classifications for the test cases.
|
||||
|
||||
Once the prototype demonstrates reliable classification, we can proceed to evaluate its performance and plan the integration into the main Asset Processor Tool.
|
||||
@ -29,7 +29,7 @@
|
||||
],
|
||||
"map_type_mapping": [
|
||||
{
|
||||
"target_type": "COL",
|
||||
"target_type": "MAP_COL",
|
||||
"keywords": [
|
||||
"COLOR*",
|
||||
"COL",
|
||||
@ -40,7 +40,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "NRM",
|
||||
"target_type": "MAP_NRM",
|
||||
"keywords": [
|
||||
"NORMAL*",
|
||||
"NORM*",
|
||||
@ -49,27 +49,27 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"target_type": "MAP_ROUGH",
|
||||
"keywords": [
|
||||
"ROUGHNESS",
|
||||
"ROUGH"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"target_type": "MAP_GLOSS",
|
||||
"keywords": [
|
||||
"GLOSS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "AO",
|
||||
"target_type": "MAP_AO",
|
||||
"keywords": [
|
||||
"AMBIENTOCCLUSION",
|
||||
"AO"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "DISP",
|
||||
"target_type": "MAP_DISP",
|
||||
"keywords": [
|
||||
"DISPLACEMENT",
|
||||
"DISP",
|
||||
@ -78,7 +78,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "REFL",
|
||||
"target_type": "MAP_REFL",
|
||||
"keywords": [
|
||||
"REFLECTION",
|
||||
"REFL",
|
||||
@ -87,26 +87,26 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "SSS",
|
||||
"target_type": "MAP_SSS",
|
||||
"keywords": [
|
||||
"SSS",
|
||||
"SUBSURFACE*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "FUZZ",
|
||||
"target_type": "MAP_FUZZ",
|
||||
"keywords": [
|
||||
"FUZZ"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "IDMAP",
|
||||
"target_type": "MAP_IDMAP",
|
||||
"keywords": [
|
||||
"IDMAP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "MASK",
|
||||
"target_type": "MAP_MASK",
|
||||
"keywords": [
|
||||
"OPAC*",
|
||||
"TRANSP*",
|
||||
@ -115,7 +115,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "METAL",
|
||||
"target_type": "MAP_METAL",
|
||||
"keywords": [
|
||||
"METAL*",
|
||||
"METALLIC"
|
||||
|
||||
@ -25,102 +25,10 @@
|
||||
"*.pdf",
|
||||
"*.url",
|
||||
"*.htm*",
|
||||
"*_Fabric.*"
|
||||
],
|
||||
"map_type_mapping": [
|
||||
{
|
||||
"target_type": "COL",
|
||||
"keywords": [
|
||||
"COLOR*",
|
||||
"COL",
|
||||
"DIFFUSE",
|
||||
"DIF",
|
||||
"ALBEDO"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "NRM",
|
||||
"keywords": [
|
||||
"NORMAL*",
|
||||
"NORM*",
|
||||
"NRM*",
|
||||
"N"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"keywords": [
|
||||
"ROUGHNESS",
|
||||
"ROUGH"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"keywords": [
|
||||
"GLOSS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "AO",
|
||||
"keywords": [
|
||||
"AMBIENTOCCLUSION",
|
||||
"AO"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "DISP",
|
||||
"keywords": [
|
||||
"DISPLACEMENT",
|
||||
"DISP",
|
||||
"HEIGHT",
|
||||
"BUMP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "REFL",
|
||||
"keywords": [
|
||||
"REFLECTION",
|
||||
"REFL",
|
||||
"SPECULAR",
|
||||
"SPEC"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "SSS",
|
||||
"keywords": [
|
||||
"SSS",
|
||||
"SUBSURFACE*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "FUZZ",
|
||||
"keywords": [
|
||||
"FUZZ"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "IDMAP",
|
||||
"keywords": [
|
||||
"IDMAP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "MASK",
|
||||
"keywords": [
|
||||
"OPAC*",
|
||||
"TRANSP*",
|
||||
"MASK*",
|
||||
"ALPHA*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "METAL",
|
||||
"keywords": [
|
||||
"METAL*",
|
||||
"METALLIC"
|
||||
]
|
||||
}
|
||||
"*_Fabric.*",
|
||||
"*_Albedo*"
|
||||
],
|
||||
"map_type_mapping": [],
|
||||
"asset_category_rules": {
|
||||
"model_patterns": [
|
||||
"*.fbx",
|
||||
|
||||
@ -29,7 +29,7 @@
|
||||
],
|
||||
"map_type_mapping": [
|
||||
{
|
||||
"target_type": "COL",
|
||||
"target_type": "MAP_COL",
|
||||
"keywords": [
|
||||
"COLOR*",
|
||||
"COL",
|
||||
@ -39,7 +39,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "NRM",
|
||||
"target_type": "MAP_NRM",
|
||||
"keywords": [
|
||||
"NORMAL*",
|
||||
"NORM*",
|
||||
@ -48,27 +48,27 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"target_type": "MAP_ROUGH",
|
||||
"keywords": [
|
||||
"ROUGHNESS",
|
||||
"ROUGH"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "ROUGH",
|
||||
"target_type": "MAP_ROUGH",
|
||||
"keywords": [
|
||||
"GLOSS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "AO",
|
||||
"target_type": "MAP_AO",
|
||||
"keywords": [
|
||||
"AMBIENTOCCLUSION",
|
||||
"AO"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "DISP",
|
||||
"target_type": "MAP_DISP",
|
||||
"keywords": [
|
||||
"DISPLACEMENT",
|
||||
"DISP",
|
||||
@ -77,7 +77,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "REFL",
|
||||
"target_type": "MAP_REFL",
|
||||
"keywords": [
|
||||
"REFLECTION",
|
||||
"REFL",
|
||||
@ -86,27 +86,27 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "SSS",
|
||||
"target_type": "MAP_SSS",
|
||||
"keywords": [
|
||||
"SSS",
|
||||
"SUBSURFACE*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "FUZZ",
|
||||
"target_type": "MAP_FUZZ",
|
||||
"keywords": [
|
||||
"FUZZ"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "IDMAP",
|
||||
"target_type": "MAP_IDMAP",
|
||||
"keywords": [
|
||||
"ID*",
|
||||
"IDMAP"
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "MASK",
|
||||
"target_type": "MAP_MASK",
|
||||
"keywords": [
|
||||
"OPAC*",
|
||||
"TRANS*",
|
||||
@ -115,7 +115,7 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"target_type": "METAL",
|
||||
"target_type": "MAP_METAL",
|
||||
"keywords": [
|
||||
"METALNESS_",
|
||||
"METALLIC"
|
||||
|
||||
@ -1,106 +0,0 @@
|
||||
# DRAFT README Enhancements - Architecture Section & Refinements
|
||||
|
||||
**(Note: This is a draft. Integrate the "Architecture" section and the refinements into the main `readme.md` file.)**
|
||||
|
||||
---
|
||||
|
||||
## Refinements to Existing Sections
|
||||
|
||||
**(Suggest adding these points or similar wording to the relevant existing sections)**
|
||||
|
||||
* **In Features:**
|
||||
* Add: **Responsive GUI:** Utilizes background threads for processing and file preview generation, ensuring the user interface remains responsive.
|
||||
* Add: **Optimized Classification:** Pre-compiles regular expressions from presets for faster file identification during classification.
|
||||
* **In Directory Structure:**
|
||||
* Update Core Logic bullet: `* **Core Logic:** main.py, monitor.py, asset_processor.py, configuration.py, config.py` (explicitly add `configuration.py`).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
**(Suggest adding this new section, perhaps after "Features" or "Directory Structure")**
|
||||
|
||||
This section provides a higher-level overview of the tool's internal structure and design, intended for developers or users interested in the technical implementation.
|
||||
|
||||
### Core Components
|
||||
|
||||
The tool is primarily built around several key Python modules:
|
||||
|
||||
* **`config.py`**: Defines core, global settings (output paths, resolutions, default behaviors, format rules, etc.) that are generally not supplier-specific.
|
||||
* **`Presets/*.json`**: Supplier-specific JSON files defining rules for interpreting source assets (filename patterns, map type keywords, model identification, etc.).
|
||||
* **`configuration.py` (`Configuration` class)**: Responsible for loading the core `config.py` settings and merging them with a selected preset JSON file. Crucially, it also **pre-compiles** regular expression patterns defined in the preset (e.g., for map keywords, extra files, 16-bit variants) upon initialization. This pre-compilation significantly speeds up the file classification process.
|
||||
* **`asset_processor.py` (`AssetProcessor` class)**: Contains the core logic for processing a *single* asset. It orchestrates the pipeline steps: workspace setup, extraction, file classification, metadata determination, map processing, channel merging, metadata file generation, and output organization.
|
||||
* **`main.py`**: Serves as the entry point for the Command-Line Interface (CLI). It handles argument parsing, sets up logging, manages the parallel processing pool, and calls `AssetProcessor` for each input asset via a wrapper function.
|
||||
* **`gui/`**: Contains modules related to the Graphical User Interface (GUI), built using PySide6.
|
||||
* **`monitor.py`**: Implements the directory monitoring functionality for automated processing.
|
||||
|
||||
### Parallel Processing (CLI & GUI)
|
||||
|
||||
To accelerate the processing of multiple assets, the tool utilizes Python's `concurrent.futures.ProcessPoolExecutor`.
|
||||
|
||||
* Both `main.py` (for CLI) and `gui/processing_handler.py` (for GUI background tasks) create a process pool.
|
||||
* The actual processing for each asset is delegated to the `main.process_single_asset_wrapper` function. This wrapper is executed in a separate worker process within the pool.
|
||||
* The wrapper function is responsible for instantiating the `Configuration` and `AssetProcessor` classes for the specific asset being processed in that worker. This isolates each asset's processing environment.
|
||||
* Results (success, skip, failure, error messages) are communicated back from the worker processes to the main coordinating script (either `main.py` or `gui/processing_handler.py`).
|
||||
|
||||
### Asset Processing Pipeline (`AssetProcessor` class)
|
||||
|
||||
The `AssetProcessor` class executes a sequence of steps for each asset:
|
||||
|
||||
1. **`_setup_workspace()`**: Creates a temporary directory for processing.
|
||||
2. **`_extract_input()`**: Extracts the input ZIP archive or copies the input folder contents into the temporary workspace.
|
||||
3. **`_inventory_and_classify_files()`**: This is a critical step that scans the workspace and classifies each file based on rules defined in the loaded `Configuration` (which includes the preset). It uses the pre-compiled regex patterns for efficiency. Key logic includes:
|
||||
* Identifying files explicitly marked for the `Extra/` folder.
|
||||
* Identifying model files.
|
||||
* Matching potential texture maps against keyword patterns.
|
||||
* Identifying and prioritizing 16-bit variants (e.g., `_NRM16.tif`) over their 8-bit counterparts based on `source_naming.bit_depth_variants` patterns. Ignored 8-bit files are tracked.
|
||||
* Handling map variants (e.g., multiple Color maps) by assigning suffixes (`-1`, `-2`) based on the `RESPECT_VARIANT_MAP_TYPES` setting in `config.py` and the order of keywords defined in the preset's `map_type_mapping`.
|
||||
* Classifying any remaining files as 'Unrecognised' (which are also moved to the `Extra/` folder).
|
||||
4. **`_determine_base_metadata()`**: Determines the asset's base name, category (Texture, Asset, Decal), and archetype (e.g., Wood, Metal) based on classified files and preset rules (`source_naming`, `asset_category_rules`, `archetype_rules`).
|
||||
5. **Skip Check**: If `overwrite` is false, checks if the final output directory and metadata file already exist. If so, processing for this asset stops early.
|
||||
6. **`_process_maps()`**: Iterates through classified texture maps. For each map:
|
||||
* Loads the image data (handling potential Gloss->Roughness inversion).
|
||||
* Resizes the map to each target resolution specified in `config.py`, avoiding upscaling.
|
||||
* Determines the output bit depth based on `MAP_BIT_DEPTH_RULES` (`respect` source or `force_8bit`).
|
||||
* Determines the output file format (`.jpg`, `.png`, `.exr`) based on a combination of factors:
|
||||
* The `RESOLUTION_THRESHOLD_FOR_JPG` (forces JPG for 8-bit maps above the threshold).
|
||||
* The original input file format (e.g., `.jpg` inputs tend to produce `.jpg` outputs if 8-bit and below threshold).
|
||||
* The target bit depth (16-bit outputs use configured `OUTPUT_FORMAT_16BIT_PRIMARY` or `_FALLBACK`).
|
||||
* Configured 8-bit format (`OUTPUT_FORMAT_8BIT`).
|
||||
* Saves the processed map for each resolution, applying appropriate compression/quality settings. Includes fallback logic if saving in the primary format fails (e.g., EXR -> PNG).
|
||||
* Calculates basic image statistics (Min/Max/Mean) for a reference resolution (`CALCULATE_STATS_RESOLUTION`).
|
||||
7. **`_merge_maps()`**: Combines channels from different processed maps into new textures (e.g., NRMRGH) based on `MAP_MERGE_RULES` defined in `config.py`. It determines the output format for merged maps similarly to `_process_maps`, considering the formats of the input maps involved.
|
||||
8. **`_generate_metadata_file()`**: Collects all gathered information (asset name, maps present, resolutions, stats, etc.) and writes it to the `metadata.json` file.
|
||||
9. **`_organize_output_files()`**: Moves the processed maps, merged maps, models, metadata file, and any 'Extra'/'Unrecognised'/'Ignored' files from the temporary workspace to the final structured output directory (`<output_base>/<supplier>/<asset_name>/`).
|
||||
10. **`_cleanup_workspace()`**: Removes the temporary workspace directory.
|
||||
|
||||
### GUI Architecture (`gui/`)
|
||||
|
||||
The GUI provides an interactive way to use the tool and manage presets.
|
||||
|
||||
* **Framework**: Built using `PySide6`, the official Python bindings for the Qt framework.
|
||||
* **Main Window (`main_window.py`)**: Defines the main application window, which includes:
|
||||
* An integrated preset editor panel (using `QSplitter`).
|
||||
* A processing panel with drag-and-drop support, a file preview table, and processing controls.
|
||||
* **Threading Model**: To prevent the UI from freezing during potentially long operations, background tasks are run in separate `QThread`s:
|
||||
* **`ProcessingHandler` (`processing_handler.py`)**: Manages the execution of the main processing pipeline (using `ProcessPoolExecutor` and `main.process_single_asset_wrapper`, similar to the CLI) in a background thread.
|
||||
* **`PredictionHandler` (`prediction_handler.py`)**: Manages the generation of file previews in a background thread. It calls `AssetProcessor.get_detailed_file_predictions()`, which performs the extraction and classification steps without full image processing, making it much faster.
|
||||
* **Communication**: Qt's **signal and slot mechanism** is used for communication between the background threads (`ProcessingHandler`, `PredictionHandler`) and the main GUI thread (`MainWindow`). For example, signals are emitted to update the progress bar, populate the preview table, and report completion status or errors.
|
||||
* **Preset Editor**: The editor allows creating, modifying, and saving preset JSON files directly within the GUI. Changes are tracked, and users are prompted to save before closing or loading another preset if changes are pending.
|
||||
|
||||
### Monitor Architecture (`monitor.py`)
|
||||
|
||||
The `monitor.py` script enables automated processing of assets dropped into a designated input directory.
|
||||
|
||||
* **File System Watching**: Uses the `watchdog` library (specifically `PollingObserver` for cross-platform compatibility) to monitor the specified `INPUT_DIR`.
|
||||
* **Event Handling**: A custom `ZipHandler` detects `on_created` events for `.zip` files.
|
||||
* **Filename Parsing**: It expects filenames in the format `[preset]_filename.zip` and uses a regular expression (`PRESET_FILENAME_REGEX`) to extract the `preset` name.
|
||||
* **Preset Validation**: Checks if the extracted preset name corresponds to a valid `.json` file in the `Presets/` directory.
|
||||
* **Processing Trigger**: If the filename format and preset are valid, it calls the `main.run_processing` function (the same core logic used by the CLI) to process the detected ZIP file using the extracted preset.
|
||||
* **File Management**: Moves the source ZIP file to either a `PROCESSED_DIR` (on success/skip) or an `ERROR_DIR` (on failure or invalid preset) after the processing attempt.
|
||||
|
||||
### Error Handling
|
||||
|
||||
* Custom exception classes (`ConfigurationError`, `AssetProcessingError`) are defined and used to signal specific types of errors during configuration loading or asset processing.
|
||||
* Standard Python logging is used throughout the application (CLI, GUI, Monitor, Core Logic) to record information, warnings, and errors. Log levels can be configured.
|
||||
* Worker processes in the processing pool capture exceptions and report them back to the main process for logging and status updates.
|
||||
@ -1,124 +0,0 @@
|
||||
# Blender Integration Plan: Node Groups from Processed Assets
|
||||
|
||||
**Objective:** Develop a Python script (`blenderscripts/create_nodegroups.py`) to run manually inside Blender. This script will scan the output directory generated by the Asset Processor Tool, read `metadata.json` files, and create/update corresponding PBR node groups in the active Blender file, leveraging the pre-calculated metadata.
|
||||
|
||||
**Key Principles:**
|
||||
|
||||
* **Leverage Existing Tool Output:** Rely entirely on the structured output and `metadata.json` from the Asset Processor Tool. Avoid reprocessing or recalculating data already available.
|
||||
* **Blender Environment:** The script is designed solely for Blender's Python environment (`bpy`).
|
||||
* **Manual Execution:** Users will manually run this script from Blender's Text Editor.
|
||||
* **Target Active File:** All operations modify the currently open `.blend` file.
|
||||
* **Assume Templates:** The script will assume node group templates (`Template_PBRSET`, `Template_PBRTYPE`) exist in the active file. Error handling will be added if they are missing.
|
||||
* **Focus on Node Groups:** The script's scope is limited to creating and updating the node groups, not materials.
|
||||
|
||||
**Detailed Plan:**
|
||||
|
||||
1. **Script Setup & Configuration:**
|
||||
* Create a new Python file named `create_nodegroups.py` (intended location: `blenderscripts/`).
|
||||
* Import necessary modules (`bpy`, `os`, `json`, `pathlib`).
|
||||
* Define user-configurable variables at the top:
|
||||
* `PROCESSED_ASSET_LIBRARY_ROOT`: Path to the root output directory of the Asset Processor Tool.
|
||||
* `PARENT_TEMPLATE_NAME`: Name of the parent node group template (e.g., `"Template_PBRSET"`).
|
||||
* `CHILD_TEMPLATE_NAME`: Name of the child node group template (e.g., `"Template_PBRTYPE"`).
|
||||
* `ASPECT_RATIO_NODE_LABEL`: Label of the Value node in the parent template for aspect ratio correction (e.g., `"AspectRatioCorrection"`).
|
||||
* `STATS_NODE_PREFIX`: Prefix for Combine XYZ nodes storing stats in the parent template (e.g., `"Histogram-"`).
|
||||
* `ENABLE_MANIFEST`: Boolean flag to enable/disable the manifest system (default: `True`).
|
||||
|
||||
2. **Manifest Handling:**
|
||||
* **Location:** Use a separate JSON file named `[ActiveBlendFileName]_manifest.json`, located in the same directory as the active `.blend` file.
|
||||
* **Loading:** Implement `load_manifest(context)` that finds and reads the manifest JSON file. If not found or invalid, return an empty dictionary.
|
||||
* **Saving:** Implement `save_manifest(context, manifest_data)` that writes the `manifest_data` dictionary to the manifest JSON file.
|
||||
* **Checking:** Implement helper functions `is_asset_processed(manifest_data, asset_name)` and `is_map_processed(manifest_data, asset_name, map_type, resolution)` to check against the loaded manifest.
|
||||
* **Updating:** Update the manifest dictionary in memory as assets/maps are processed. Save the manifest once at the end of the script.
|
||||
|
||||
3. **Core Logic - `process_library()` function:**
|
||||
* Get Blender context.
|
||||
* Load manifest data (if enabled).
|
||||
* Validate that `PROCESSED_ASSET_LIBRARY_ROOT` exists.
|
||||
* Validate that template node groups exist in `bpy.data.node_groups`. Exit gracefully with an error message if not found.
|
||||
* Initialize counters (new groups, updated groups, etc.).
|
||||
* **Scan Directory:** Use `os.walk` or `pathlib.rglob` to find all `metadata.json` files within the `PROCESSED_ASSET_LIBRARY_ROOT`.
|
||||
* **Iterate Metadata:** For each `metadata.json` found:
|
||||
* Parse the JSON data. Extract key information: `asset_name`, `supplier_name`, `archetype`, `maps` (dictionary of maps with resolutions, paths, stats), `aspect_ratio_change_string`.
|
||||
* Check manifest if asset is already processed (if enabled). Skip if true.
|
||||
* **Parent Group Handling:**
|
||||
* Determine target parent group name (e.g., `f"PBRSET_{asset_name}"`).
|
||||
* Find existing group or create a copy from `PARENT_TEMPLATE_NAME`.
|
||||
* Mark group as asset (`asset_mark()`) if not already.
|
||||
* **Apply Metadata:**
|
||||
* Find the aspect ratio node using `ASPECT_RATIO_NODE_LABEL`. Calculate the correction factor based on the `aspect_ratio_change_string` from metadata (using helper function `calculate_factor_from_string`) and set the node's default value.
|
||||
* For relevant map types (e.g., ROUGH, DISP), find the stats node (`STATS_NODE_PREFIX` + map type). Set the X, Y, Z inputs using the `min`, `max`, `mean` values stored in the map's metadata entry for the reference resolution.
|
||||
* **Apply Asset Tags:** Use `asset_data.tags.new()` to add the `supplier_name` and `archetype` tags (checking for existence first).
|
||||
* **Child Group Handling (Iterate through `maps` in metadata):**
|
||||
* For each `map_type` (e.g., "COL", "NRM") and its data in the metadata:
|
||||
* Determine target child group name (e.g., `f"PBRTYPE_{asset_name}_{map_type}"`).
|
||||
* Find existing child group or create a copy from `CHILD_TEMPLATE_NAME`.
|
||||
* Find the corresponding placeholder node in the *parent* group (by label matching `map_type`). Assign the child node group to this placeholder (`placeholder_node.node_tree = child_group`).
|
||||
* Link the child group's output to the corresponding parent group's output socket. Ensure the parent output socket type is `NodeSocketColor`.
|
||||
* **Image Node Handling (Iterate through resolutions for the map type):**
|
||||
* For each `resolution` (e.g., "4K", "2K") and its `image_path` in the metadata:
|
||||
* Check manifest if this specific map/resolution is processed (if enabled). Skip if true.
|
||||
* Find the corresponding Image Texture node in the *child* group (by label matching `resolution`, e.g., "4K").
|
||||
* Load the image using `bpy.data.images.load(image_path, check_existing=True)`. Handle potential file-not-found errors.
|
||||
* Assign the loaded image to the `image_node.image`.
|
||||
* Set the `image_node.image.colorspace_settings.name` based on the `map_type` (using a helper function `get_color_space`).
|
||||
* Update manifest dictionary for this map/resolution (if enabled).
|
||||
* Update manifest dictionary for the processed asset (if enabled).
|
||||
* Save manifest data (if enabled and changes were made).
|
||||
* Print summary (duration, groups created/updated, etc.).
|
||||
|
||||
4. **Helper Functions:**
|
||||
* `find_nodes_by_label(node_tree, label, node_type)`: Reusable function to find nodes.
|
||||
* `calculate_factor_from_string(aspect_string)`: Parses the `aspect_ratio_change_string` from metadata and returns the appropriate UV X-scaling factor.
|
||||
* `get_color_space(map_type)`: Returns the appropriate Blender color space name for a given map type string.
|
||||
* `add_tag_if_new(asset_data, tag_name)`: Adds a tag if it doesn't exist.
|
||||
* Manifest loading/saving/checking functions.
|
||||
|
||||
5. **Execution Block (`if __name__ == "__main__":`)**
|
||||
* Add pre-run checks (templates exist, library path valid, blend file saved if manifest enabled).
|
||||
* Call the main `process_library()` function.
|
||||
* Include basic timing and print statements for start/end.
|
||||
|
||||
**Mermaid Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start Script in Blender] --> B(Load Config: Lib Path, Template Names, Node Labels);
|
||||
B --> C{Check Templates Exist};
|
||||
C -- Templates OK --> D(Load Manifest from adjacent .json file);
|
||||
C -- Templates Missing --> X(Error & Exit);
|
||||
D --> E{Scan Processed Library for metadata.json};
|
||||
E --> F{For each metadata.json};
|
||||
F --> G{Parse Metadata (Asset Name, Supplier, Archetype, Maps, Aspect Str, Stats)};
|
||||
G --> H{Is Asset in Manifest?};
|
||||
H -- Yes --> F;
|
||||
H -- No --> I{Find/Create Parent Group (PBRSET_)};
|
||||
I --> J(Mark as Asset & Apply Supplier + Archetype Tags);
|
||||
J --> K(Find Aspect Node & Set Value from Aspect String);
|
||||
K --> M{For each Map Type in Metadata};
|
||||
M --> N(Find Stats Node & Set Values from Stats in Metadata);
|
||||
N --> O{Find/Create Child Group (PBRTYPE_)};
|
||||
O --> P(Assign Child to Parent Placeholder);
|
||||
P --> Q(Link Child Output to Parent Output);
|
||||
Q --> R{For each Resolution of Map};
|
||||
R --> S{Is Map/Res in Manifest?};
|
||||
S -- Yes --> R;
|
||||
S -- No --> T(Find Image Node in Child);
|
||||
T --> U(Load Processed Image);
|
||||
U --> V(Assign Image to Node);
|
||||
V --> W(Set Image Color Space);
|
||||
W --> W1(Update Manifest Dict for Map/Res);
|
||||
W1 --> R;
|
||||
R -- All Resolutions Done --> M;
|
||||
M -- All Map Types Done --> X1(Update Manifest Dict for Asset);
|
||||
X1 --> F;
|
||||
F -- All metadata.json Processed --> Y(Save Manifest Dict to adjacent .json file);
|
||||
Y --> Z(Print Summary & Finish);
|
||||
|
||||
subgraph "Manifest Operations (External File)"
|
||||
D; H; S; W1; X1; Y;
|
||||
end
|
||||
|
||||
subgraph "Node/Asset Operations"
|
||||
I; J; K; N; O; P; Q; T; U; V; W;
|
||||
end
|
||||
@ -1,52 +0,0 @@
|
||||
# Blender Integration Plan v2
|
||||
|
||||
## Goal
|
||||
|
||||
Add an optional step to `main.py` to run `blenderscripts/create_nodegroups.py` and `blenderscripts/create_materials.py` on specified `.blend` files after asset processing is complete.
|
||||
|
||||
## Proposed Plan
|
||||
|
||||
1. **Update `config.py`:**
|
||||
* Add two new optional configuration variables: `DEFAULT_NODEGROUP_BLEND_PATH` and `DEFAULT_MATERIALS_BLEND_PATH`. These will store the default paths to the Blender files.
|
||||
|
||||
2. **Update `main.py` Argument Parser:**
|
||||
* Add two new optional command-line arguments: `--nodegroup-blend` and `--materials-blend`.
|
||||
* These arguments will accept file paths to the respective `.blend` files.
|
||||
* If provided, these arguments will override the default paths specified in `config.py`.
|
||||
|
||||
3. **Update `blenderscripts/create_nodegroups.py` and `blenderscripts/create_materials.py`:**
|
||||
* Modify both scripts to accept the processed asset library root path (`PROCESSED_ASSET_LIBRARY_ROOT`) as a command-line argument. This will be passed to the script when executed by Blender using the `--` separator.
|
||||
* Update the scripts to read this path from `sys.argv` instead of using the hardcoded variable.
|
||||
|
||||
4. **Update `main.py` Execution Flow:**
|
||||
* After the main asset processing loop (`run_processing`) completes and the summary is reported, check if the `--nodegroup-blend` or `--materials-blend` arguments (or their fallbacks from `config.py`) were provided.
|
||||
* If a path for the nodegroup `.blend` file is available:
|
||||
* Construct a command to execute Blender in the background (`-b`), load the specified nodegroup `.blend` file, run the `create_nodegroups.py` script using `--python`, pass the processed asset root directory as an argument after `--`, and save the `.blend` file (`-S`).
|
||||
* Execute this command using the `execute_command` tool.
|
||||
* If a path for the materials `.blend` file is available:
|
||||
* Construct a similar command to execute Blender in the background, load the specified materials `.blend` file, run the `create_materials.py` script using `--python`, pass the processed asset root directory as an argument after `--`, and save the `.blend` file (`-S`).
|
||||
* Execute this command using the `execute_command` tool.
|
||||
* Include error handling for the execution of the Blender commands.
|
||||
|
||||
## Execution Flow Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Asset Processing Complete] --> B[Report Summary];
|
||||
B --> C{Nodegroup Blend Path Specified?};
|
||||
C -- Yes --> D[Get Nodegroup Blend Path (Arg or Config)];
|
||||
D --> E[Construct Blender Command for Nodegroups];
|
||||
E --> F[Execute Command: blender -b nodegroup.blend --python create_nodegroups.py -- <asset_root> -S];
|
||||
F --> G{Command Successful?};
|
||||
G -- Yes --> H{Materials Blend Path Specified?};
|
||||
G -- No --> I[Log Nodegroup Error];
|
||||
I --> H;
|
||||
H -- Yes --> J[Get Materials Blend Path (Arg or Config)];
|
||||
J --> K[Construct Blender Command for Materials];
|
||||
K --> L[Execute Command: blender -b materials.blend --python create_materials.py -- <asset_root> -S];
|
||||
L --> M{Command Successful?};
|
||||
M -- Yes --> N[End main.py];
|
||||
M -- No --> O[Log Materials Error];
|
||||
O --> N;
|
||||
H -- No --> N;
|
||||
C -- No --> H;
|
||||
@ -1,131 +0,0 @@
|
||||
# Blender Material Creation Script Plan
|
||||
|
||||
This document outlines the plan for creating a new Blender script (`create_materials.py`) in the `blenderscripts/` directory. This script will scan the processed asset library output by the Asset Processor Tool, read the `metadata.json` files, and create or update Blender materials that link to the corresponding PBRSET node groups found in a specified Blender Asset Library. The script will also set the material's viewport properties using pre-calculated statistics from the metadata. The script will skip processing an asset if the corresponding material already exists in the current Blender file.
|
||||
|
||||
## 1. Script Location and Naming
|
||||
|
||||
* Create a new file: `blenderscripts/create_materials.py`.
|
||||
|
||||
## 2. Script Structure
|
||||
|
||||
The script will follow a similar structure to `blenderscripts/create_nodegroups.py`, including:
|
||||
|
||||
* Import statements (`bpy`, `os`, `json`, `pathlib`, `time`, `base64`).
|
||||
* A `--- USER CONFIGURATION ---` section at the top.
|
||||
* Helper functions.
|
||||
* A main processing function (e.g., `process_library_for_materials`).
|
||||
* An execution block (`if __name__ == "__main__":`) to run the main function.
|
||||
|
||||
## 3. Configuration Variables
|
||||
|
||||
The script will include the following configuration variables in the `--- USER CONFIGURATION ---` section:
|
||||
|
||||
* `PROCESSED_ASSET_LIBRARY_ROOT`: Path to the root output directory of the Asset Processor Tool (same as in `create_nodegroups.py`). This is used to find the `metadata.json` files and reference images for previews.
|
||||
* `PBRSET_ASSET_LIBRARY_NAME`: The name of the Blender Asset Library (configured in Blender Preferences) that contains the PBRSET node groups created by `create_nodegroups.py`.
|
||||
* `TEMPLATE_MATERIAL_NAME`: Name of the required template material in the Blender file (e.g., "Template_PBRMaterial").
|
||||
* `PLACEHOLDER_NODE_LABEL`: Label of the placeholder Group node within the template material's node tree where the PBRSET node group will be linked (e.g., "PBRSET_PLACEHOLDER").
|
||||
* `MATERIAL_NAME_PREFIX`: Prefix for the created materials (e.g., "Mat_").
|
||||
* `PBRSET_GROUP_PREFIX`: Prefix used for the PBRSET node groups created by `create_nodegroups.py` (e.g., "PBRSET_").
|
||||
* `REFERENCE_MAP_TYPES`: List of map types to look for to find a reference image for the material preview (e.g., `["COL", "COL-1"]`).
|
||||
* `REFERENCE_RESOLUTION_ORDER`: Preferred resolution order for the reference image (e.g., `["1K", "512", "2K", "4K"]`).
|
||||
* `IMAGE_FILENAME_PATTERN`: Assumed filename pattern for processed images (same as in `create_nodegroups.py`).
|
||||
* `FALLBACK_IMAGE_EXTENSIONS`: Fallback extensions for finding image files (same as in `create_nodegroups.py`).
|
||||
* `VIEWPORT_COLOR_MAP_TYPES`: List of map types to check in metadata's `image_stats_1k` for viewport diffuse color.
|
||||
* `VIEWPORT_ROUGHNESS_MAP_TYPES`: List of map types to check in metadata's `image_stats_1k` for viewport roughness.
|
||||
* `VIEWPORT_METALLIC_MAP_TYPES`: List of map types to check in metadata's `image_stats_1k` for viewport metallic.
|
||||
|
||||
## 4. Helper Functions
|
||||
|
||||
The script will include the following helper functions:
|
||||
|
||||
* `find_nodes_by_label(node_tree, label, node_type=None)`: Reusable from `create_nodegroups.py` to find nodes in a node tree.
|
||||
* `add_tag_if_new(asset_data, tag_name)`: Reusable from `create_nodegroups.py` to add asset tags.
|
||||
* `reconstruct_image_path_with_fallback(...)`: Reusable from `create_nodegroups.py` to find image paths (needed for setting the custom preview).
|
||||
* `get_stat_value(stats_dict, map_type_list, stat_key)`: A helper function to safely retrieve a specific statistic from the `image_stats_1k` dictionary.
|
||||
|
||||
## 5. Main Processing Logic (`process_library_for_materials`)
|
||||
|
||||
The main function will perform the following steps:
|
||||
|
||||
* **Pre-run Checks:**
|
||||
* Verify `PROCESSED_ASSET_LIBRARY_ROOT` exists and is a directory.
|
||||
* Verify the `PBRSET_ASSET_LIBRARY_NAME` exists in Blender's user preferences (`bpy.context.preferences.filepaths.asset_libraries`).
|
||||
* Verify the `TEMPLATE_MATERIAL_NAME` material exists and uses nodes.
|
||||
* Verify the `PLACEHOLDER_NODE_LABEL` Group node exists in the template material's node tree.
|
||||
* **Scan for Metadata:**
|
||||
* Iterate through supplier directories within `PROCESSED_ASSET_LIBRARY_ROOT`.
|
||||
* Iterate through asset directories within each supplier directory.
|
||||
* Identify `metadata.json` files.
|
||||
* **Process Each Metadata File:**
|
||||
* Load the `metadata.json` file.
|
||||
* Extract `asset_name`, `supplier_name`, `archetype`, `processed_map_resolutions`, `merged_map_resolutions`, `map_details`, and `image_stats_1k`.
|
||||
* Determine the expected PBRSET node group name: `f"{PBRSET_GROUP_PREFIX}{asset_name}"`.
|
||||
* Determine the target material name: `f"{MATERIAL_NAME_PREFIX}{asset_name}"`.
|
||||
* **Find or Create Material:**
|
||||
* Check if a material with the `target_material_name` already exists in `bpy.data.materials`.
|
||||
* If it exists, log a message indicating the asset is being skipped and move to the next metadata file.
|
||||
* If it doesn't exist, copy the `TEMPLATE_MATERIAL_NAME` material and rename the copy to `target_material_name` (create mode). Handle potential copy failures.
|
||||
* **Find Placeholder Node:**
|
||||
* Find the node with `PLACEHOLDER_NODE_LABEL` in the target material's node tree using `find_nodes_by_label`. Handle cases where the node is not found or is not a Group node.
|
||||
* **Find and Link PBRSET Node Group from Asset Library:**
|
||||
* Get the path to the `.blend` file associated with the `PBRSET_ASSET_LIBRARY_NAME` from user preferences.
|
||||
* Use `bpy.data.libraries.load(filepath, link=True)` to link the node group with `target_pbrset_group_name` from the external `.blend` file into the current file. Handle cases where the library or the node group is not found.
|
||||
* Once linked, get the reference to the newly linked node group in `bpy.data.node_groups`.
|
||||
* **Link Linked Node Group to Placeholder:**
|
||||
* If both the placeholder node and the *newly linked* PBRSET node group are found, assign the linked node group to the `node_tree` property of the placeholder node.
|
||||
* **Mark Material as Asset:**
|
||||
* If the material is new or not already marked, call `material.asset_mark()`.
|
||||
* **Copy Asset Tags:**
|
||||
* If both the material and the *linked* PBRSET node group have asset data, copy tags (supplier, archetype) from the node group to the material using `add_tag_if_new`.
|
||||
* **Set Custom Preview:**
|
||||
* Find a suitable reference image path (e.g., lowest resolution COL map) using `reconstruct_image_path_with_fallback` and the `REFERENCE_MAP_TYPES` and `REFERENCE_RESOLUTION_ORDER` configurations.
|
||||
* If a reference image path is found, use `bpy.ops.ed.lib_id_load_custom_preview` to set the custom preview for the material. This operation requires overriding the context.
|
||||
* **Set Viewport Properties (using metadata stats):**
|
||||
* Check if `image_stats_1k` is present and valid in the metadata.
|
||||
* **Diffuse Color:** Use the `get_stat_value` helper to get the 'mean' stat for map types in `VIEWPORT_COLOR_MAP_TYPES`. If found and is a valid color list `[R, G, B]`, set `material.diffuse_color` to this value.
|
||||
* **Roughness:** Use the `get_stat_value` helper to get the 'mean' stat for map types in `VIEWPORT_ROUGHNESS_MAP_TYPES`. If found, get the first value (for grayscale). Check if stats for `VIEWPORT_METALLIC_MAP_TYPES` exist. If metallic stats are *not* found, invert the roughness value (`1.0 - value`) before assigning it to `material.roughness`. Clamp the final value between 0.0 and 1.0.
|
||||
* **Metallic:** Use the `get_stat_value` helper to get the 'mean' stat for map types in `VIEWPORT_METALLIC_MAP_TYPES`. If found, get the first value (for grayscale) and assign it to `material.metallic`. If metallic stats are *not* found, set `material.metallic` to 0.0.
|
||||
* **Error Handling and Reporting:**
|
||||
* Include `try...except` blocks to catch errors during file reading, JSON parsing, Blender operations, linking, etc.
|
||||
* Print informative messages about progress, creation/update status, and errors.
|
||||
* **Summary Report:**
|
||||
* Print a summary of how many metadata files were processed, materials created/updated, node groups linked, errors encountered, and assets skipped.
|
||||
|
||||
## 6. Process Flow Diagram (Updated)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start Script] --> B{Pre-run Checks Pass?};
|
||||
B -- No --> C[Abort Script];
|
||||
B -- Yes --> D[Scan Processed Asset Root];
|
||||
D --> E{Found metadata.json?};
|
||||
E -- No --> F[Finish Script (No Assets)];
|
||||
E -- Yes --> G[Loop through metadata.json files];
|
||||
G --> H[Read metadata.json];
|
||||
H --> I{Metadata Valid?};
|
||||
I -- No --> J[Log Error, Skip Asset];
|
||||
I -- Yes --> K[Extract Asset Info & Stats];
|
||||
K --> L[Find or Create Material];
|
||||
L --> M{Material Exists?};
|
||||
M -- Yes --> N[Log Skip, Continue Loop];
|
||||
M -- No --> O[Find Placeholder Node in Material];
|
||||
O --> P[Find PBRSET NG in Library & Link];
|
||||
P --> Q{Linking Successful?};
|
||||
Q -- No --> R[Log Error, Skip Asset];
|
||||
Q -- Yes --> S[Link Linked NG to Placeholder];
|
||||
S --> T[Mark Material as Asset];
|
||||
T --> U[Copy Asset Tags];
|
||||
U --> V[Find Reference Image Path for Preview];
|
||||
V --> W{Reference Image Found?};
|
||||
W -- Yes --> X[Set Custom Material Preview];
|
||||
W -- No --> Y[Log Warning (No Preview)];
|
||||
X --> Z[Set Viewport Properties from Stats];
|
||||
Y --> Z;
|
||||
Z --> AA[Increment Counters];
|
||||
AA --> G;
|
||||
G --> AB[Print Summary Report];
|
||||
AB --> AC[End Script];
|
||||
|
||||
J --> G;
|
||||
N --> G;
|
||||
R --> G;
|
||||
@ -1,103 +0,0 @@
|
||||
# Blender Addon Plan: Material Merger
|
||||
|
||||
**Version:** 1.1 (Includes Extensibility Consideration)
|
||||
|
||||
**1. Goal:**
|
||||
Create a standalone Blender addon that allows users to select two existing materials (generated by the Asset Processor Tool, or previously merged by this addon) and merge them into a new material. The merge should preserve their individual node structures (including custom tweaks) and combine their final outputs using a dedicated `MaterialMerge` node group.
|
||||
|
||||
**2. Core Functionality (Approach 2 - Node Copying):**
|
||||
|
||||
* **Trigger:** User selects two materials in Blender and invokes an operator (e.g., via a button in the Shader Editor's UI panel).
|
||||
* **New Material Creation:** The addon creates a new Blender material, named appropriately (e.g., `MAT_Merged_<NameA>_<NameB>`).
|
||||
* **Node Copying:**
|
||||
* For *each* selected source material:
|
||||
* Iterate through its node tree.
|
||||
* Copy all nodes *except* the `Material Output` node into the *new* material's node tree, attempting to preserve relative layout and offsetting subsequent copies.
|
||||
* **Identify Final Outputs:** Determine the node providing the final BSDF shader output and the node providing the final Displacement output *before* the original `Material Output` node.
|
||||
* In a base material (from Asset Processor), these are expected to be the `PBR_BSDF` node group (BSDF output) and the `PBR_Handler` node group (Displacement output).
|
||||
* In an already-merged material, these will be the outputs of its top-level `MaterialMerge` node group.
|
||||
* Store references to these final output nodes and their relevant sockets.
|
||||
* **MaterialMerge Node:**
|
||||
* **Link/Append** the `MaterialMerge` node group into the new material's node tree.
|
||||
* **Assumption:** This node group exists in `blender_files/utility_nodegroups.blend` relative to the addon's location.
|
||||
* **Assumption:** Socket names are `Shader A`, `Shader B`, `Displacement A`, `Displacement B` (inputs) and `BSDF`, `Displacement` (outputs).
|
||||
* **Connections:**
|
||||
* Connect the identified final BSDF output of the *first* source material's copied structure to the `MaterialMerge` node's `Shader A` input.
|
||||
* Connect the identified final Displacement output of the *first* source material's copied structure to the `MaterialMerge` node's `Displacement A` input.
|
||||
* Connect the identified final BSDF output of the *second* source material's copied structure to the `MaterialMerge` node's `Shader B` input.
|
||||
* Connect the identified final Displacement output of the *second* source material's copied structure to the `MaterialMerge` node's `Displacement B` input.
|
||||
* Connect the `MaterialMerge` node's `BSDF` output to the new material's `Material Output` node's `Surface` input.
|
||||
* Connect the `MaterialMerge` node's `Displacement` output to the new material's `Material Output` node's `Displacement` input.
|
||||
* **Layout:** Optionally, attempt a basic auto-layout (`node_tree.nodes.update()`) or arrange the key nodes logically.
|
||||
|
||||
**3. User Interface (UI):**
|
||||
|
||||
* A simple panel in the Blender Shader Editor (Properties region - 'N' panel).
|
||||
* Two dropdowns or search fields allowing the user to select existing materials from the current `.blend` file.
|
||||
* A button labeled "Merge Selected Materials".
|
||||
* Status messages/feedback (e.g., "Merged material created: [Name]", "Error: Could not find required nodes in [Material Name]").
|
||||
|
||||
**4. Addon Structure (Python):**
|
||||
|
||||
* `__init__.py`: Registers the addon, panel, and operator classes.
|
||||
* `operator.py`: Contains the `OT_MergeMaterials` operator class implementing the core logic.
|
||||
* `panel.py`: Contains the `PT_MaterialMergePanel` class defining the UI layout.
|
||||
* (Optional) `utils.py`: Helper functions for node finding, copying, linking, identifying final outputs, etc.
|
||||
|
||||
**5. Error Handling:**
|
||||
|
||||
* Check if two valid materials are selected.
|
||||
* Verify that the selected materials have node trees.
|
||||
* Handle cases where the expected final BSDF/Displacement output nodes cannot be reliably identified in one or both source materials.
|
||||
* Handle potential errors during node copying.
|
||||
* Handle errors if the `utility_nodegroups.blend` file or the `MaterialMerge` node group within it cannot be found/linked.
|
||||
|
||||
**6. Assumptions to Verify (Based on User Feedback):**
|
||||
|
||||
* **Node Identification:**
|
||||
* Base Material Handler: Node named `PBR_Handler`.
|
||||
* Base Material BSDF: Node named `PBR_BSDF`.
|
||||
* Merged Material Outputs: The `BSDF` and `Displacement` outputs of the top-level `MaterialMerge` node.
|
||||
* **`MaterialMerge` Node:**
|
||||
* Location: `blender_files/utility_nodegroups.blend` (relative path).
|
||||
* Input Sockets: `Shader A`, `Shader B`, `Displacement A`, `Displacement B`.
|
||||
* Output Sockets: `BSDF`, `Displacement`.
|
||||
|
||||
**7. Future Extensibility - Recursive Merging:**
|
||||
|
||||
* The core merging logic (copying nodes, identifying final outputs, connecting to a new `MaterialMerge` node) is designed to inherently support selecting an already-merged material as an input without requiring separate code paths initially. The identification of final BSDF/Displacement outputs needs to correctly handle both base materials and merged materials (checking for `PBR_BSDF`/`PBR_Handler` or the outputs of an existing `MaterialMerge` node).
|
||||
|
||||
**8. Mermaid Diagram of Node Flow:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph New Merged Material
|
||||
subgraph Copied from Source A (Mat_A or Merge_A)
|
||||
%% Nodes representing the structure of Source A
|
||||
Structure_A[...]
|
||||
Final_BSDF_A[Final BSDF Output A]
|
||||
Final_Disp_A[Final Displacement Output A]
|
||||
Structure_A --> Final_BSDF_A
|
||||
Structure_A --> Final_Disp_A
|
||||
end
|
||||
|
||||
subgraph Copied from Source B (Mat_B or Merge_B)
|
||||
%% Nodes representing the structure of Source B
|
||||
Structure_B[...]
|
||||
Final_BSDF_B[Final BSDF Output B]
|
||||
Final_Disp_B[Final Displacement Output B]
|
||||
Structure_B --> Final_BSDF_B
|
||||
Structure_B --> Final_Disp_B
|
||||
end
|
||||
|
||||
Merge[MaterialMerge]
|
||||
Output[Material Output]
|
||||
|
||||
Final_BSDF_A -- BSDF --> Merge -- Shader A --> Merge
|
||||
Final_Disp_A -- Displacement --> Merge -- Displacement A --> Merge
|
||||
Final_BSDF_B -- BSDF --> Merge -- Shader B --> Merge
|
||||
Final_Disp_B -- Displacement --> Merge -- Displacement B --> Merge
|
||||
|
||||
Merge -- BSDF --> Output -- Surface --> Output
|
||||
Merge -- Displacement --> Output -- Displacement --> Output
|
||||
end
|
||||
107
ProjectNotes/ConfigurationRefactoringPlan.md
Normal file
107
ProjectNotes/ConfigurationRefactoringPlan.md
Normal file
@ -0,0 +1,107 @@
|
||||
# Configuration System Refactoring Plan
|
||||
|
||||
This document outlines the plan for refactoring the configuration system of the Asset Processor Tool.
|
||||
|
||||
## Overall Goals
|
||||
|
||||
1. **Decouple Definitions:** Separate `ASSET_TYPE_DEFINITIONS` and `FILE_TYPE_DEFINITIONS` from the main `config/app_settings.json` into dedicated files.
|
||||
2. **Introduce User Overrides:** Allow users to override base settings via a new `config/user_settings.json` file.
|
||||
3. **Improve GUI Saving:** (Lower Priority) Make GUI configuration saving more targeted to avoid overwriting unrelated settings when saving changes from `ConfigEditorDialog` or `LLMEditorWidget`.
|
||||
|
||||
## Proposed Plan Phases
|
||||
|
||||
**Phase 1: Decouple Definitions**
|
||||
|
||||
1. **Create New Definition Files:**
|
||||
* Create `config/asset_type_definitions.json`.
|
||||
* Create `config/file_type_definitions.json`.
|
||||
2. **Migrate Content:**
|
||||
* Move `ASSET_TYPE_DEFINITIONS` object from `config/app_settings.json` to `config/asset_type_definitions.json`.
|
||||
* Move `FILE_TYPE_DEFINITIONS` object from `config/app_settings.json` to `config/file_type_definitions.json`.
|
||||
3. **Update `configuration.py`:**
|
||||
* Add constants for new definition file paths.
|
||||
* Modify `Configuration` class to load these new files.
|
||||
* Update property methods (e.g., `get_asset_type_definitions`, `get_file_type_definitions_with_examples`) to use data from the new definition dictionaries.
|
||||
* Adjust validation (`_validate_configs`) as needed.
|
||||
4. **Update GUI & `load_base_config()`:**
|
||||
* Modify `load_base_config()` to load and return a combined dictionary including `app_settings.json` and the two new definition files.
|
||||
* Update GUI components relying on `load_base_config()` to ensure they receive the necessary definition data.
|
||||
|
||||
**Phase 2: Implement User Overrides**
|
||||
|
||||
1. **Define `user_settings.json`:**
|
||||
* Establish `config/user_settings.json` for user-specific overrides, mirroring parts of `app_settings.json`.
|
||||
2. **Update `configuration.py` Loading:**
|
||||
* In `Configuration.__init__`, load `app_settings.json`, then definition files, then attempt to load and deep merge `user_settings.json` (user settings override base).
|
||||
* Load presets *after* the base+user merge (presets override combined base+user).
|
||||
* Modify `load_base_config()` to also load and merge `user_settings.json` after `app_settings.json`.
|
||||
3. **Update GUI Editors:**
|
||||
* Modify `ConfigEditorDialog` to load the effective settings (base+user) but save changes *only* to `config/user_settings.json`.
|
||||
* `LLMEditorWidget` continues targeting `llm_settings.json`.
|
||||
|
||||
**Phase 3: Granular GUI Saving (Lower Priority)**
|
||||
|
||||
1. **Refactor Saving Logic:**
|
||||
* In `ConfigEditorDialog` and `LLMEditorWidget`:
|
||||
* Load the current target file (`user_settings.json` or `llm_settings.json`).
|
||||
* Identify specific setting(s) changed by the user in the GUI session.
|
||||
* Update only those specific key(s) in the loaded dictionary.
|
||||
* Write the entire modified dictionary back to the target file, preserving untouched settings.
|
||||
|
||||
## Proposed File Structure & Loading Flow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Config Files
|
||||
A[config/asset_type_definitions.json]
|
||||
B[config/file_type_definitions.json]
|
||||
C[config/app_settings.json (Base Defaults)]
|
||||
D[config/user_settings.json (User Overrides)]
|
||||
E[config/llm_settings.json]
|
||||
F[config/suppliers.json]
|
||||
G[Presets/*.json]
|
||||
end
|
||||
|
||||
subgraph Code
|
||||
H[configuration.py]
|
||||
I[GUI]
|
||||
J[Processing Engine / Pipeline]
|
||||
K[LLM Handlers]
|
||||
end
|
||||
|
||||
subgraph Loading Flow (Configuration Class)
|
||||
L(Load Asset Types) --> H
|
||||
M(Load File Types) --> H
|
||||
N(Load Base Settings) --> P(Merge Base + User)
|
||||
O(Load User Settings) --> P
|
||||
P --> R(Merge Preset Overrides)
|
||||
Q(Load LLM Settings) --> H
|
||||
R --> T(Final Config Object)
|
||||
G -- Load Preset --> R
|
||||
H -- Contains --> T
|
||||
end
|
||||
|
||||
subgraph Loading Flow (GUI - load_base_config)
|
||||
L2(Load Asset Types) --> U(Return Merged Defaults + Defs)
|
||||
M2(Load File Types) --> U
|
||||
N2(Load Base Settings) --> V(Merge Base + User)
|
||||
O2(Load User Settings) --> V
|
||||
V --> U
|
||||
I -- Calls --> U
|
||||
end
|
||||
|
||||
|
||||
T -- Used by --> J
|
||||
T -- Used by --> K
|
||||
|
||||
I -- Edits --> D
|
||||
I -- Edits --> E
|
||||
I -- Manages --> F
|
||||
|
||||
style A fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style B fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style C fill:#ccf,stroke:#333,stroke-width:2px
|
||||
style D fill:#9cf,stroke:#333,stroke-width:2px
|
||||
style E fill:#ccf,stroke:#333,stroke-width:2px
|
||||
style F fill:#9cf,stroke:#333,stroke-width:2px
|
||||
style G fill:#ffc,stroke:#333,stroke-width:2px
|
||||
@ -1,118 +0,0 @@
|
||||
# Data Interface: GUI Preview Edits to Processing Handler
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the data structures and interface used to pass user edits made in the GUI's file preview table (specifically changes to 'Status' and 'Predicted Asset' output name) to the backend `ProcessingHandler`. It also incorporates a structure for future asset-level properties.
|
||||
|
||||
## 2. Data Structures
|
||||
|
||||
Two primary data structures are used:
|
||||
|
||||
### 2.1. File Data (`file_list`)
|
||||
|
||||
* **Type:** `list[dict]`
|
||||
* **Description:** A flat list where each dictionary represents a single file identified during the prediction phase. This list is modified by the GUI to reflect user edits to file status and output names.
|
||||
* **Dictionary Keys (per file):**
|
||||
* `original_path` (`str`): The full path to the source file. (Read-only by GUI)
|
||||
* `predicted_asset_name` (`str | None`): The name of the asset group the file belongs to, derived from input. (Read-only by GUI)
|
||||
* `predicted_output_name` (`str | None`): The backend's predicted final output filename. **EDITABLE** by the user in the GUI ('Predicted Asset' column).
|
||||
* `status` (`str`): The backend's predicted status (e.g., 'Mapped', 'Ignored'). **EDITABLE** by the user in the GUI ('Status' column).
|
||||
* `details` (`str | None`): Additional information or error messages. (Read-only by GUI, potentially updated by model validation).
|
||||
* `source_asset` (`str`): An identifier for the source asset group (e.g., input folder/zip name). (Read-only by GUI)
|
||||
|
||||
### 2.2. Asset Properties (`asset_properties`)
|
||||
|
||||
* **Type:** `dict[str, dict]`
|
||||
* **Description:** A dictionary mapping the `source_asset` identifier (string key) to a dictionary of asset-level properties determined by the backend prediction/preset. This structure is initially read-only in the GUI but designed for future expansion (e.g., editing asset category).
|
||||
* **Asset Properties Dictionary Keys (Example):**
|
||||
* `asset_category` (`str`): The determined category (e.g., 'surface', 'model').
|
||||
* `asset_tags` (`list[str]`): Any relevant tags associated with the asset.
|
||||
* *(Other future asset-level properties can be added here)*
|
||||
|
||||
## 3. Data Flow & Interface
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Backend
|
||||
A[Prediction Logic] -- Generates --> B(file_list);
|
||||
A -- Generates --> C(asset_properties);
|
||||
end
|
||||
|
||||
subgraph GUI Components
|
||||
D[PredictionHandler] -- prediction_results_ready(file_list, asset_properties) --> E(PreviewTableModel);
|
||||
E -- Stores & Allows Edits --> F(Internal file_list);
|
||||
E -- Stores --> G(Internal asset_properties);
|
||||
H[MainWindow] -- Retrieves --> F;
|
||||
H -- Retrieves --> G;
|
||||
H -- Passes (file_list, asset_properties) --> I[ProcessingHandler];
|
||||
end
|
||||
|
||||
subgraph Backend
|
||||
I -- Uses Edited --> F;
|
||||
I -- Uses Read-Only --> G;
|
||||
end
|
||||
|
||||
style A fill:#lightblue,stroke:#333,stroke-width:1px
|
||||
style B fill:#lightgreen,stroke:#333,stroke-width:1px
|
||||
style C fill:#lightyellow,stroke:#333,stroke-width:1px
|
||||
style D fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style E fill:#ccf,stroke:#333,stroke-width:2px
|
||||
style F fill:#lightgreen,stroke:#333,stroke-width:1px
|
||||
style G fill:#lightyellow,stroke:#333,stroke-width:1px
|
||||
style H fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style I fill:#lightblue,stroke:#333,stroke-width:2px
|
||||
|
||||
```
|
||||
|
||||
1. **Prediction:** The `PredictionHandler` generates both the `file_list` and `asset_properties`.
|
||||
2. **Signal:** It emits a signal (e.g., `prediction_results_ready`) containing both structures, likely as a tuple `(file_list, asset_properties)`.
|
||||
3. **Table Model:** The `PreviewTableModel` receives the tuple. It stores `asset_properties` (read-only for now). It stores `file_list` and allows user edits to the `status` and `predicted_output_name` values within this list.
|
||||
4. **Processing Trigger:** When the user initiates processing, the `MainWindow` retrieves the (potentially modified) `file_list` and the (unmodified) `asset_properties` from the `PreviewTableModel`.
|
||||
5. **Processing Execution:** The `MainWindow` passes both structures to the `ProcessingHandler`.
|
||||
6. **Handler Logic:** The `ProcessingHandler` iterates through the `file_list`. For each file, it uses the potentially edited `status` and `predicted_output_name`. If asset-level information is needed, it uses the file's `source_asset` key to look up the data in the `asset_properties` dictionary.
|
||||
|
||||
## 4. Example Data Passed to `ProcessingHandler`
|
||||
|
||||
* **`file_list` (Example):**
|
||||
```python
|
||||
[
|
||||
{
|
||||
'original_path': 'C:/Path/To/AssetA/AssetA_Diffuse.png',
|
||||
'predicted_asset_name': 'AssetA',
|
||||
'predicted_output_name': 'T_AssetA_BC.tga',
|
||||
'status': 'Ignored', # <-- User Edit
|
||||
'details': 'User override: Ignored',
|
||||
'source_asset': 'AssetA'
|
||||
},
|
||||
{
|
||||
'original_path': 'C:/Path/To/AssetA/AssetA_Normal.png',
|
||||
'predicted_asset_name': 'AssetA',
|
||||
'predicted_output_name': 'T_AssetA_Normals_DX.tga', # <-- User Edit
|
||||
'status': 'Mapped',
|
||||
'details': None,
|
||||
'source_asset': 'AssetA'
|
||||
},
|
||||
# ... other files
|
||||
]
|
||||
```
|
||||
* **`asset_properties` (Example):**
|
||||
```python
|
||||
{
|
||||
'AssetA': {
|
||||
'asset_category': 'surface',
|
||||
'asset_tags': ['wood', 'painted']
|
||||
},
|
||||
'AssetB': {
|
||||
'asset_category': 'model',
|
||||
'asset_tags': ['metal', 'sci-fi']
|
||||
}
|
||||
# ... other assets
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Implications
|
||||
|
||||
* **`PredictionHandler`:** Needs modification to generate and emit both `file_list` and `asset_properties`. Signal signature changes.
|
||||
* **`PreviewTableModel`:** Needs modification to receive, store, and provide both structures. Must implement editing capabilities (`flags`, `setData`) for the relevant columns using the `file_list`. Needs methods like `get_edited_data()` returning both structures.
|
||||
* **`MainWindow`:** Needs modification to retrieve both structures from the table model and pass them to the `ProcessingHandler`.
|
||||
* **`ProcessingHandler`:** Needs modification to accept both structures in its processing method signature. Must update logic to use the edited `status` and `predicted_output_name` from `file_list` and look up data in `asset_properties` using `source_asset` when needed.
|
||||
@ -1,37 +0,0 @@
|
||||
# FEAT-003: Selective Nodegroup Generation and Category Tagging - Implementation Plan
|
||||
|
||||
**Objective:** Modify `blenderscripts/create_nodegroups.py` to read the asset category from `metadata.json`, conditionally create nodegroups for "Surface" and "Decal" assets, and add the category as a tag to the Blender asset.
|
||||
|
||||
**Plan:**
|
||||
|
||||
1. **Modify `blenderscripts/create_nodegroups.py`:**
|
||||
* Locate the main loop in `blenderscripts/create_nodegroups.py` that iterates through the processed assets.
|
||||
* Inside this loop, for each asset directory, construct the path to the `metadata.json` file.
|
||||
* Read the `metadata.json` file using Python's `json` module.
|
||||
* Extract the `category` value from the parsed JSON data.
|
||||
* Implement a conditional check: If the extracted `category` is *not* "Surface" and *not* "Decal", skip the existing nodegroup creation logic for this asset and proceed to the tagging step.
|
||||
* If the `category` *is* "Surface" or "Decal", execute the existing nodegroup creation logic.
|
||||
* After the conditional nodegroup creation (or skipping), use the Blender Python API (`bpy`) to find the corresponding Blender asset (likely the created node group, if applicable, or potentially the asset representation even if the nodegroup was skipped).
|
||||
* Add the extracted `category` string as a tag to the found Blender asset.
|
||||
|
||||
2. **Testing:**
|
||||
* Prepare a test set of processed assets that includes examples of "Surface", "Decal", and "Asset" categories, each with a corresponding `metadata.json` file.
|
||||
* Run the modified `create_nodegroups.py` script within Blender, pointing it to the test asset library root.
|
||||
* Verify in Blender that:
|
||||
* Node groups were created only for the "Surface" and "Decal" assets.
|
||||
* No node groups were created for "Asset" category assets.
|
||||
* All processed assets (Surface, Decal, and Asset) have a tag corresponding to their category ("Surface", "Decal", or "Asset") in the Blender asset browser.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start Script] --> B{Iterate Assets};
|
||||
B --> C[Read metadata.json];
|
||||
C --> D[Get Category];
|
||||
D --> E{Category in ["Surface", "Decal"]?};
|
||||
E -- Yes --> F[Create Nodegroup];
|
||||
E -- No --> G[Skip Nodegroup];
|
||||
F --> H[Find Blender Asset];
|
||||
G --> H[Find Blender Asset];
|
||||
H --> I[Add Category Tag];
|
||||
I --> B;
|
||||
B -- No More Assets --> J[End Script];
|
||||
@ -1,49 +0,0 @@
|
||||
# Plan for Adding .rar and .7z Support
|
||||
|
||||
**Goal:** Extend the Asset Processor Tool to accept `.rar` and `.7z` files as input sources, in addition to the currently supported `.zip` files and folders.
|
||||
|
||||
**Plan:**
|
||||
|
||||
1. **Add Required Libraries:**
|
||||
* Update the `requirements.txt` file to include `py7zr` and `rarfile` as dependencies. This will ensure these libraries are installed when setting up the project.
|
||||
|
||||
2. **Modify Input Extraction Logic:**
|
||||
* Locate the `_extract_input` method within the `AssetProcessor` class in `asset_processor.py`.
|
||||
* Modify this method to check the file extension of the input source.
|
||||
* If the extension is `.zip`, retain the existing extraction logic using Python's built-in `zipfile` module.
|
||||
* If the extension is `.rar`, implement extraction using the `rarfile` library.
|
||||
* If the extension is `.7z`, implement extraction using the `py7zr` library.
|
||||
* Include error handling for cases where the archive might be corrupted, encrypted (since we are not implementing password support at this stage, these should likely be skipped or logged as errors), or uses an unsupported compression method. Log appropriate warnings or errors in such cases.
|
||||
* If the input is a directory, retain the existing logic to copy its contents to the temporary workspace.
|
||||
|
||||
3. **Update CLI and Monitor Input Handling:**
|
||||
* Review `main.py` (CLI entry point) and `monitor.py` (Directory Monitor).
|
||||
* Ensure that the argument parsing in `main.py` can accept `.rar` and `.7z` file paths as valid inputs.
|
||||
* In `monitor.py`, modify the `ZipHandler` (or create a new handler) to watch for `.rar` and `.7z` file creation events in the watched directory, in addition to `.zip` files. The logic for triggering processing via `main.run_processing` should then be extended to handle these new file types.
|
||||
|
||||
4. **Update Documentation:**
|
||||
* Edit `Documentation/00_Overview.md` to explicitly mention `.rar` and `.7z` as supported input formats in the overview section.
|
||||
* Update `Documentation/01_User_Guide/02_Features.md` to list `.rar` and `.7z` alongside `.zip` and folders in the features list.
|
||||
* Modify `Documentation/01_User_Guide/03_Installation.md` to include instructions for installing the new `py7zr` and `rarfile` dependencies (likely via `pip install -r requirements.txt`).
|
||||
* Revise `Documentation/02_Developer_Guide/05_Processing_Pipeline.md` to accurately describe the updated `_extract_input` method, detailing how `.zip`, `.rar`, `.7z`, and directories are handled.
|
||||
|
||||
5. **Testing:**
|
||||
* Prepare sample `.rar` and `.7z` files (including nested directories and various file types) to test the extraction logic thoroughly.
|
||||
* Test processing of these new archive types via both the CLI and the Directory Monitor.
|
||||
* Verify that the subsequent processing steps (classification, map processing, metadata generation, etc.) work correctly with files extracted from `.rar` and `.7z` archives.
|
||||
|
||||
Here is a simplified flow diagram illustrating the updated input handling:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Input Source] --> B{Is it a file or directory?};
|
||||
B -- Directory --> C[Copy Contents to Workspace];
|
||||
B -- File --> D{What is the file extension?};
|
||||
D -- .zip --> E[Extract using zipfile];
|
||||
D -- .rar --> F[Extract using rarfile];
|
||||
D -- .7z --> G[Extract using py7zr];
|
||||
E --> H[Temporary Workspace];
|
||||
F --> H;
|
||||
G --> H;
|
||||
C --> H;
|
||||
H --> I[Processing Pipeline Starts];
|
||||
@ -1,51 +0,0 @@
|
||||
# Plan: Implement "Force Lossless" Format for Specific Map Types
|
||||
|
||||
**Goal:** Modify the asset processor to ensure specific map types ("NRM", "DISP") are always saved in a lossless format (PNG or EXR based on bit depth), overriding the JPG threshold and input format rules. This rule should apply to both individually processed maps and merged maps.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. **Add Configuration Setting (`config.py`):**
|
||||
* Introduce a new list named `FORCE_LOSSLESS_MAP_TYPES` in `config.py`.
|
||||
* Populate this list: `FORCE_LOSSLESS_MAP_TYPES = ["NRM", "DISP"]`.
|
||||
|
||||
2. **Expose Setting in `Configuration` Class (`configuration.py`):**
|
||||
* Add a default value for `FORCE_LOSSLESS_MAP_TYPES` in the `_load_core_config` method's `default_core_settings` dictionary.
|
||||
* Add a new property to the `Configuration` class to access this list:
|
||||
```python
|
||||
@property
|
||||
def force_lossless_map_types(self) -> list:
|
||||
"""Gets the list of map types that must always be saved losslessly."""
|
||||
return self._core_settings.get('FORCE_LOSSLESS_MAP_TYPES', [])
|
||||
```
|
||||
|
||||
3. **Modify `_process_maps` Method (`asset_processor.py`):**
|
||||
* Locate the section determining the output format (around line 805).
|
||||
* **Before** the `if output_bit_depth == 8 and target_dim >= threshold:` check (line 811), insert the new logic:
|
||||
* Check if the current `map_type` is in `self.config.force_lossless_map_types`.
|
||||
* If yes, determine the appropriate lossless format (`png` or configured 16-bit format like `exr`) based on `output_bit_depth`, set `output_format`, `output_ext`, `save_params`, and `needs_float16` accordingly, and skip the subsequent `elif` / `else` blocks for format determination.
|
||||
* Use an `elif` for the existing JPG threshold check and the final `else` for the rule-based logic, ensuring they only run if `force_lossless` is false.
|
||||
|
||||
4. **Modify `_merge_maps` Method (`asset_processor.py`):**
|
||||
* Locate the section determining the output format for the merged map (around line 1151).
|
||||
* **Before** the `if output_bit_depth == 8 and target_dim >= threshold:` check (line 1158), insert similar logic as in step 3:
|
||||
* Check if the `output_map_type` (the type of the *merged* map) is in `self.config.force_lossless_map_types`.
|
||||
* If yes, determine the appropriate lossless format based on the merged map's `output_bit_depth`, set `output_format`, `output_ext`, `save_params`, and `needs_float16`, and skip the subsequent `elif` / `else` blocks.
|
||||
* Use `elif` and `else` for the existing threshold and hierarchy logic.
|
||||
|
||||
**Process Flow Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Format Determination (per resolution, _process_maps & _merge_maps)
|
||||
A[Start] --> B(Get map_type / output_map_type);
|
||||
B --> C{Determine output_bit_depth};
|
||||
C --> D{Is map_type in FORCE_LOSSLESS_MAP_TYPES?};
|
||||
D -- Yes --> E[Set format = Lossless (PNG/EXR based on bit_depth)];
|
||||
D -- No --> F{Is output_bit_depth == 8 AND target_dim >= threshold?};
|
||||
F -- Yes --> G[Set format = JPG];
|
||||
F -- No --> H[Set format based on input/hierarchy/rules];
|
||||
G --> I[Set Save Params];
|
||||
H --> I;
|
||||
E --> I;
|
||||
I --> J[Save Image];
|
||||
end
|
||||
@ -1,17 +0,0 @@
|
||||
Here is a summary of our session focused on adding OpenEXR support:
|
||||
|
||||
Goal:
|
||||
Enable the Asset Processor Tool to reliably write .exr files for 16-bit maps, compatible with Windows executables and Docker.
|
||||
|
||||
Approach:
|
||||
We decided to use the dedicated openexr Python library for writing EXR files, rather than compiling a custom OpenCV build. The plan involved modifying asset_processor.py to use this library, updating the Dockerfile with necessary system libraries, and outlining steps for PyInstaller on Windows.
|
||||
|
||||
Issues & Fixes:
|
||||
|
||||
Initial Changes: Modified asset_processor.py and Dockerfile.
|
||||
NameError: name 'log' is not defined: Fixed by moving the logger initialization earlier in asset_processor.py.
|
||||
AttributeError: module 'Imath' has no attribute 'Header' & NameError: name 'fallback_fmt' is not defined: Attempted fixes by changing Imath.Header to OpenEXR.Header and correcting one instance of fallback_fmt.
|
||||
TypeError: __init__(): incompatible constructor arguments... Invoked with: HALF & NameError: name 'fallback_fmt' is not defined (again): Corrected the OpenEXR.Channel constructor call and fixed the remaining fallback_fmt typo.
|
||||
concurrent.futures.process.BrokenProcessPool: The script crashed when attempting the EXR save, likely due to an internal OpenEXR library error or a Windows dependency issue.
|
||||
Next Steps (when resuming):
|
||||
Investigate the BrokenProcessPool error, possibly by testing EXR saving with a simple array or verifying Windows dependencies for the OpenEXR library.
|
||||
@ -1,52 +0,0 @@
|
||||
# GUI Blender Integration Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Add a checkbox and input fields to the GUI (`gui/main_window.py`) to enable/disable Blender script execution and specify the `.blend` file paths, defaulting to `config.py` values. Integrate this control into the processing logic (`gui/processing_handler.py`).
|
||||
|
||||
## Proposed Plan
|
||||
|
||||
1. **Modify `gui/main_window.py`:**
|
||||
* Add a `QCheckBox` (e.g., `self.blender_integration_checkbox`) to the processing panel layout.
|
||||
* Add two pairs of `QLineEdit` widgets and `QPushButton` browse buttons for the nodegroup `.blend` path (`self.nodegroup_blend_path_input`, `self.browse_nodegroup_blend_button`) and the materials `.blend` path (`self.materials_blend_path_input`, `self.browse_materials_blend_button`).
|
||||
* Initialize the text of the `QLineEdit` widgets by reading the `DEFAULT_NODEGROUP_BLEND_PATH` and `DEFAULT_MATERIALS_BLEND_PATH` values from `config.py` when the GUI starts.
|
||||
* Connect signals from the browse buttons to new methods that open a `QFileDialog` to select `.blend` files and update the corresponding input fields.
|
||||
* Modify the slot connected to the "Start Processing" button to:
|
||||
* Read the checked state of `self.blender_integration_checkbox`.
|
||||
* Read the text from `self.nodegroup_blend_path_input` and `self.materials_blend_path_input`.
|
||||
* Pass these three pieces of information (checkbox state, nodegroup path, materials path) to the `ProcessingHandler` when initiating the processing task.
|
||||
|
||||
2. **Modify `gui/processing_handler.py`:**
|
||||
* Add parameters to the method that starts the processing (likely `start_processing`) to accept the Blender integration flag (boolean), the nodegroup `.blend` path (string), and the materials `.blend` path (string).
|
||||
* Implement the logic for finding the Blender executable (reading `BLENDER_EXECUTABLE_PATH` from `config.py` or checking PATH) within `processing_handler.py`.
|
||||
* Implement the logic for executing the Blender scripts using `subprocess.run` within `processing_handler.py`. This logic should be similar to the `run_blender_script` function added to `main.py` in the previous step.
|
||||
* Ensure this Blender script execution logic is conditional based on the received integration flag and runs *after* the main asset processing (handled by the worker pool) is complete.
|
||||
|
||||
## Execution Flow Diagram (GUI)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[GUI: User Clicks Start Processing] --> B{Blender Integration Checkbox Checked?};
|
||||
B -- Yes --> C[Get Blend File Paths from Input Fields];
|
||||
C --> D[Pass Paths and Flag to ProcessingHandler];
|
||||
D --> E[ProcessingHandler: Start Asset Processing (Worker Pool)];
|
||||
E --> F[ProcessingHandler: Asset Processing Complete];
|
||||
F --> G{Blender Integration Flag True?};
|
||||
G -- Yes --> H[ProcessingHandler: Find Blender Executable];
|
||||
H --> I{Blender Executable Found?};
|
||||
I -- Yes --> J{Nodegroup Blend Path Valid?};
|
||||
J -- Yes --> K[ProcessingHandler: Run Nodegroup Script in Blender];
|
||||
K --> L{Script Successful?};
|
||||
L -- Yes --> M{Materials Blend Path Valid?};
|
||||
L -- No --> N[ProcessingHandler: Report Nodegroup Error];
|
||||
N --> M;
|
||||
M -- Yes --> O[ProcessingHandler: Run Materials Script in Blender];
|
||||
O --> P{Script Successful?};
|
||||
P -- Yes --> Q[ProcessingHandler: Report Completion];
|
||||
P -- No --> R[ProcessingHandler: Report Materials Error];
|
||||
R --> Q;
|
||||
M -- No --> Q;
|
||||
J -- No --> M[Skip Nodegroup Script];
|
||||
I -- No --> Q[Skip Blender Scripts];
|
||||
G -- No --> Q;
|
||||
B -- No --> E;
|
||||
@ -1,43 +0,0 @@
|
||||
# GUI Enhancement Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Implement two new features in the Graphical User Interface (GUI) of the Asset Processor Tool:
|
||||
1. Automatically switch the preview to "simple view" when more than 10 input files (ZIPs or folders) are added to the queue.
|
||||
2. Remove the specific visual area labeled "Drag and drop folders here" while keeping the drag-and-drop functionality active for the main processing panel.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
The changes will be made in the `gui/main_window.py` file.
|
||||
|
||||
1. **Implement automatic preview switch:**
|
||||
* Locate the `add_input_paths` method.
|
||||
* After adding the `newly_added_paths` to `self.current_asset_paths`, check the total number of items in `self.current_asset_paths`.
|
||||
* If the count is greater than 10, programmatically set the state of the "Disable Detailed Preview" menu action (`self.toggle_preview_action`) to `checked=True`. This will automatically trigger the `update_preview` method, which will then render the simple list view.
|
||||
|
||||
2. **Remove the "Drag and drop folders here" visual area:**
|
||||
* Locate the `setup_main_panel_ui` method.
|
||||
* Find the creation of the `self.drag_drop_area` QFrame and its associated QLabel (`drag_drop_label`).
|
||||
* Add a line after the creation of `self.drag_drop_area` to hide this widget (`self.drag_drop_area.setVisible(False)`). This will remove the visual box and label while keeping the drag-and-drop functionality enabled for the main window.
|
||||
|
||||
## Workflow Diagram
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[User drops files/folders] --> B{Call add_input_paths}
|
||||
B --> C[Add paths to self.current_asset_paths]
|
||||
C --> D{Count items in self.current_asset_paths}
|
||||
D{Count > 10?} -->|Yes| E[Set toggle_preview_action.setChecked(True)]
|
||||
D{Count > 10?} -->|No| F[Keep current preview state]
|
||||
E --> G[update_preview triggered]
|
||||
F --> G[update_preview triggered]
|
||||
G --> H{Check toggle_preview_action state}
|
||||
H{Checked (Simple)?} -->|Yes| I[Display simple list in preview_table]
|
||||
H{Checked (Simple)?} -->|No| J[Run PredictionHandler for detailed preview]
|
||||
J --> K[Display detailed results in preview_table]
|
||||
|
||||
L[GUI Initialization] --> M[Call setup_main_panel_ui]
|
||||
M --> N[Create drag_drop_area QFrame]
|
||||
N --> O[Hide drag_drop_area QFrame]
|
||||
O --> P[Main window accepts drops]
|
||||
P --> B
|
||||
@ -1,103 +0,0 @@
|
||||
# GUI Feature Enhancement Plan
|
||||
|
||||
**Overall Goal:** Modify the GUI (`gui/main_window.py`, `gui/prediction_handler.py`) to make the output path configurable, improve UI responsiveness during preview generation, and add a toggle to switch between detailed file preview and a simple input path list.
|
||||
|
||||
**Detailed Plan:**
|
||||
|
||||
1. **Feature: Configurable Output Path**
|
||||
* **File:** `gui/main_window.py`
|
||||
* **Changes:**
|
||||
* **UI Addition:**
|
||||
* Below the `preset_combo` layout, add a new `QHBoxLayout`.
|
||||
* Inside this layout, add:
|
||||
* A `QLabel` with text "Output Directory:".
|
||||
* A `QLineEdit` (e.g., `self.output_path_edit`) to display/edit the path. Make it read-only initially if preferred, or editable.
|
||||
* A `QPushButton` (e.g., `self.browse_output_button`) with text "Browse...".
|
||||
* **Initialization (`__init__` or `setup_main_panel_ui`):**
|
||||
* Read the default `OUTPUT_BASE_DIR` from `core_config`.
|
||||
* Resolve this path relative to the project root (`project_root / output_base_dir_config`).
|
||||
* Set the initial text of `self.output_path_edit` to this resolved default path.
|
||||
* **Browse Button Logic:**
|
||||
* Connect the `clicked` signal of `self.browse_output_button` to a new method (e.g., `_browse_for_output_directory`).
|
||||
* Implement `_browse_for_output_directory`:
|
||||
* Use `QFileDialog.getExistingDirectory` to let the user select a folder.
|
||||
* If a directory is selected, update the text of `self.output_path_edit`.
|
||||
* **Processing Logic (`start_processing`):**
|
||||
* Instead of reading/resolving the path from `core_config`, get the path string directly from `self.output_path_edit.text()`.
|
||||
* Convert this string to a `Path` object.
|
||||
* **Add Validation:** Before passing the path to the handler, check if the directory exists. If not, attempt to create it using `output_dir.mkdir(parents=True, exist_ok=True)`. Handle potential `OSError` exceptions during creation and show an error message if it fails. Also, consider adding a basic writability check if possible.
|
||||
* Pass the validated `output_dir_str` to `self.processing_handler.run_processing`.
|
||||
|
||||
2. **Feature: Responsive UI (Address Prediction Bottleneck)**
|
||||
* **File:** `gui/prediction_handler.py`
|
||||
* **Changes:**
|
||||
* **Import:** Add `from concurrent.futures import ThreadPoolExecutor, as_completed`.
|
||||
* **Modify `run_prediction`:**
|
||||
* Inside the `try` block (after loading `config`), create a `ThreadPoolExecutor` (e.g., `with ThreadPoolExecutor(max_workers=...) as executor:`). Determine a reasonable `max_workers` count (e.g., `os.cpu_count() // 2` or a fixed number like 4 or 8).
|
||||
* Instead of iterating through `input_paths` sequentially, submit a task to the executor for each `input_path_str`.
|
||||
* The task submitted should be a helper method (e.g., `_predict_single_asset`) that takes `input_path_str` and the loaded `config` object as arguments.
|
||||
* `_predict_single_asset` will contain the logic currently inside the loop: instantiate `AssetProcessor`, call `get_detailed_file_predictions`, handle exceptions, and return the list of prediction dictionaries for that *single* asset (or an error dictionary).
|
||||
* Store the `Future` objects returned by `executor.submit`.
|
||||
* Use `as_completed(futures)` to process results as they become available.
|
||||
* Append the results from each completed future to the `all_file_results` list.
|
||||
* Emit `prediction_results_ready` once at the very end with the complete `all_file_results` list.
|
||||
* **File:** `gui/main_window.py`
|
||||
* **Changes:**
|
||||
* No changes needed in the `on_prediction_results_ready` slot itself, as the handler will still emit the full list at the end.
|
||||
|
||||
3. **Feature: Preview Toggle**
|
||||
* **File:** `gui/main_window.py`
|
||||
* **Changes:**
|
||||
* **UI Addition:**
|
||||
* Add a `QCheckBox` (e.g., `self.disable_preview_checkbox`) with text "Disable Detailed Preview". Place it logically, perhaps near the `overwrite_checkbox` or above the `preview_table`. Set its default state to unchecked.
|
||||
* **Modify `update_preview`:**
|
||||
* At the beginning of the method, check `self.disable_preview_checkbox.isChecked()`.
|
||||
* **If Checked (Simple View):**
|
||||
* Clear the `preview_table`.
|
||||
* Set simplified table headers (e.g., `self.preview_table.setColumnCount(1); self.preview_table.setHorizontalHeaderLabels(["Input Path"])`). Adjust column resize modes.
|
||||
* Iterate through `self.current_asset_paths`. For each path, add a row to the table containing just the path string.
|
||||
* Set status bar message (e.g., "Preview disabled. Showing input list.").
|
||||
* **Crucially:** `return` from the method here to prevent the `PredictionHandler` from being started.
|
||||
* **If Unchecked (Detailed View):**
|
||||
* Ensure the table headers and column count are set back to the detailed view configuration (Status, Original Path, Predicted Name, Details).
|
||||
* Continue with the rest of the existing `update_preview` logic to start the `PredictionHandler`.
|
||||
* **Connect Signal:** In `__init__` or `setup_main_panel_ui`, connect the `toggled` signal of `self.disable_preview_checkbox` to the `self.update_preview` slot.
|
||||
* **Initial State:** Ensure the first call to `update_preview` (if any) respects the initial unchecked state of the checkbox.
|
||||
|
||||
**Mermaid Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph MainWindow
|
||||
A[User Action: Add Asset / Change Preset / Toggle Preview] --> B{Update Preview Triggered};
|
||||
B --> C{Is 'Disable Preview' Checked?};
|
||||
C -- Yes --> D[Show Simple List View in Table];
|
||||
C -- No --> E[Set Detailed Table Headers];
|
||||
E --> F[Start PredictionHandler Thread];
|
||||
F --> G[PredictionHandler Runs];
|
||||
G --> H[Slot: Populate Table with Detailed Results];
|
||||
|
||||
I[User Clicks Start Processing] --> J{Get Output Path from UI LineEdit};
|
||||
J --> K[Validate/Create Output Path];
|
||||
K -- Path OK --> L[Start ProcessingHandler Thread];
|
||||
K -- Path Error --> M[Show Error Message];
|
||||
L --> N[ProcessingHandler Runs];
|
||||
N --> O[Update UI (Progress, Status)];
|
||||
|
||||
P[User Clicks Browse...] --> Q[Show QFileDialog];
|
||||
Q --> R[Update Output Path LineEdit];
|
||||
end
|
||||
|
||||
subgraph PredictionHandler [Background Thread]
|
||||
style PredictionHandler fill:#f9f,stroke:#333,stroke-width:2px
|
||||
F --> S{Use ThreadPoolExecutor};
|
||||
S --> T[Run _predict_single_asset Concurrently];
|
||||
T --> U[Collect Results];
|
||||
U --> V[Emit prediction_results_ready (Full List)];
|
||||
V --> H;
|
||||
end
|
||||
|
||||
subgraph ProcessingHandler [Background Thread]
|
||||
style ProcessingHandler fill:#ccf,stroke:#333,stroke-width:2px
|
||||
L --> N;
|
||||
end
|
||||
@ -1,78 +0,0 @@
|
||||
# GUI Log Console Feature Plan
|
||||
|
||||
**Overall Goal:** Add a log console panel to the GUI's editor panel, controlled by a "View" menu action. Move the "Disable Detailed Preview" control to the same "View" menu.
|
||||
|
||||
**Detailed Plan:**
|
||||
|
||||
1. **Create Custom Log Handler:**
|
||||
* **New File/Location:** Potentially add this to a new `gui/log_handler.py` or keep it within `gui/main_window.py` if simple enough.
|
||||
* **Implementation:**
|
||||
* Define a class `QtLogHandler(logging.Handler, QObject)` that inherits from both `logging.Handler` and `QObject` (for signals).
|
||||
* Add a Qt signal, e.g., `log_record_received = Signal(str)`.
|
||||
* Override the `emit(self, record)` method:
|
||||
* Format the log record using `self.format(record)`.
|
||||
* Emit the `log_record_received` signal with the formatted string.
|
||||
|
||||
2. **Modify `gui/main_window.py`:**
|
||||
* **Imports:** Add `QMenuBar`, `QMenu`, `QAction` from `PySide6.QtWidgets`. Import the new `QtLogHandler`.
|
||||
* **UI Elements (`__init__` / `setup_editor_panel_ui`):**
|
||||
* **Menu Bar:**
|
||||
* Create `self.menu_bar = self.menuBar()`.
|
||||
* Create `view_menu = self.menu_bar.addMenu("&View")`.
|
||||
* **Log Console:**
|
||||
* Create `self.log_console_output = QTextEdit()`. Set it to read-only (`self.log_console_output.setReadOnly(True)`).
|
||||
* Create a container widget, e.g., `self.log_console_widget = QWidget()`. Create a layout for it (e.g., `QVBoxLayout`) and add `self.log_console_output` to this layout.
|
||||
* In `setup_editor_panel_ui`, insert `self.log_console_widget` into the `editor_layout` *before* adding the `list_layout` (the preset list).
|
||||
* Initially hide the console: `self.log_console_widget.setVisible(False)`.
|
||||
* **Menu Actions:**
|
||||
* Create `self.toggle_log_action = QAction("Show Log Console", self, checkable=True)`. Connect `self.toggle_log_action.toggled.connect(self._toggle_log_console_visibility)`. Add it to `view_menu`.
|
||||
* Create `self.toggle_preview_action = QAction("Disable Detailed Preview", self, checkable=True)`. Connect `self.toggle_preview_action.toggled.connect(self.update_preview)`. Add it to `view_menu`.
|
||||
* **Remove Old Checkbox:** Delete the lines creating and adding `self.disable_preview_checkbox`.
|
||||
* **Logging Setup (`__init__`):**
|
||||
* Instantiate the custom handler: `self.log_handler = QtLogHandler()`.
|
||||
* Connect its signal: `self.log_handler.log_record_received.connect(self._append_log_message)`.
|
||||
* Add the handler to the logger: `log.addHandler(self.log_handler)`. Set an appropriate level if needed (e.g., `self.log_handler.setLevel(logging.INFO)`).
|
||||
* **New Slots:**
|
||||
* Implement `_toggle_log_console_visibility(self, checked)`: This slot will simply call `self.log_console_widget.setVisible(checked)`.
|
||||
* Implement `_append_log_message(self, message)`:
|
||||
* Append the `message` string to `self.log_console_output`.
|
||||
* Optional: Add logic to limit the number of lines in the text edit to prevent performance issues.
|
||||
* Optional: Add basic HTML formatting for colors based on log level.
|
||||
* **Modify `update_preview`:**
|
||||
* Replace the check for `self.disable_preview_checkbox.isChecked()` with `self.toggle_preview_action.isChecked()`.
|
||||
* Update the log messages within this method to reflect checking the action state.
|
||||
|
||||
**Mermaid Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph MainWindow
|
||||
A[Initialization] --> B(Create Menu Bar);
|
||||
B --> C(Add View Menu);
|
||||
C --> D(Add 'Show Log Console' Action);
|
||||
C --> E(Add 'Disable Detailed Preview' Action);
|
||||
A --> F(Create Log Console QTextEdit);
|
||||
F --> G(Place Log Console Widget in Layout [Hidden]);
|
||||
A --> H(Create & Add QtLogHandler);
|
||||
H --> I(Connect Log Handler Signal to _append_log_message);
|
||||
|
||||
D -- Toggled --> J[_toggle_log_console_visibility];
|
||||
J --> K(Show/Hide Log Console Widget);
|
||||
|
||||
E -- Toggled --> L[update_preview];
|
||||
|
||||
M[update_preview] --> N{Is 'Disable Preview' Action Checked?};
|
||||
N -- Yes --> O[Show Simple List View];
|
||||
N -- No --> P[Start PredictionHandler];
|
||||
|
||||
Q[Any Log Message] -- Emitted by Logger --> H;
|
||||
I --> R[_append_log_message];
|
||||
R --> S(Append Message to Log Console QTextEdit);
|
||||
end
|
||||
|
||||
subgraph QtLogHandler
|
||||
style QtLogHandler fill:#lightgreen,stroke:#333,stroke-width:2px
|
||||
T1[emit(record)] --> T2(Format Record);
|
||||
T2 --> T3(Emit log_record_received Signal);
|
||||
T3 --> I;
|
||||
end
|
||||
@ -1,119 +0,0 @@
|
||||
# Asset Processor GUI Development Plan
|
||||
|
||||
This document outlines the plan for developing a Graphical User Interface (GUI) for the Asset Processor Tool.
|
||||
|
||||
**I. Foundation & Framework Choice**
|
||||
|
||||
1. **Choose GUI Framework:** PySide6 (LGPL license, powerful features).
|
||||
2. **Project Structure:** Create `gui/` directory. Core logic remains separate.
|
||||
3. **Dependencies:** Add `PySide6` to `requirements.txt`.
|
||||
|
||||
**II. Core GUI Layout & Basic Functionality**
|
||||
|
||||
1. **Main Window:** `gui/main_window.py`.
|
||||
2. **Layout Elements:**
|
||||
* Drag & Drop Area (`QWidget` subclass).
|
||||
* Preset Dropdown (`QComboBox`).
|
||||
* Preview Area (`QListWidget`).
|
||||
* Progress Bar (`QProgressBar`).
|
||||
* Control Buttons (`QPushButton`: Start, Cancel, Manage Presets).
|
||||
* Status Bar (`QStatusBar`).
|
||||
|
||||
**III. Input Handling & Predictive Preview**
|
||||
|
||||
1. **Connect Drag & Drop:** Validate drops, add valid paths to Preview List.
|
||||
2. **Refactor for Prediction:** Create/refactor methods in `AssetProcessor`/`Configuration` to predict output names without full processing.
|
||||
3. **Implement Preview Update:** On preset change or file add, load config, call prediction logic, update Preview List items (e.g., "Input: ... -> Output: ...").
|
||||
4. **Responsive Preview:** Utilize PySide6 list widget efficiency (consider model/view for very large lists).
|
||||
|
||||
**IV. Processing Integration & Progress Reporting**
|
||||
|
||||
1. **Adapt Processing Logic:** Refactor `main.py`'s `ProcessPoolExecutor` loop into a callable function/class (`gui/processing_handler.py`).
|
||||
2. **Background Execution:** Run processing logic in a `QThread` to keep GUI responsive.
|
||||
3. **Progress Updates (Signals & Slots):**
|
||||
* Background thread emits signals: `progress_update(current, total)`, `file_status_update(path, status, msg)`, `processing_finished(stats)`.
|
||||
* Main window connects slots to these signals to update UI elements (`QProgressBar`, `QListWidget` items, status bar).
|
||||
4. **Completion/Error Handling:** Re-enable controls, display summary stats, report errors on `processing_finished`.
|
||||
|
||||
**V. Preset Management Interface (Sub-Task)**
|
||||
|
||||
1. **New Window/Dialog:** `gui/preset_editor.py`.
|
||||
2. **Functionality:** List, load, edit (tree view/form), create, save/save as presets (`.json` in `presets/`). Basic validation.
|
||||
|
||||
**VI. Refinements & Additional Features (Ideas)**
|
||||
|
||||
1. **Cancel Button:** Implement cancellation signal to background thread/workers.
|
||||
2. **Log Viewer:** Add `QTextEdit` to display `logging` output.
|
||||
3. **Output Directory Selection:** Add browse button/field.
|
||||
4. **Configuration Options:** Expose key `config.py` settings.
|
||||
5. **Clearer Error Display:** Tooltips, status bar updates, or error panel.
|
||||
|
||||
**VII. Packaging (Deployment)**
|
||||
|
||||
1. **Tooling:** PyInstaller or cx_Freeze.
|
||||
2. **Configuration:** Build scripts (`.spec`) to bundle code, dependencies, assets, and `presets/`.
|
||||
|
||||
**High-Level Mermaid Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph GUI Application (PySide6)
|
||||
A[Main Window] --> B(Drag & Drop Area);
|
||||
A --> C(Preset Dropdown);
|
||||
A --> D(Preview List Widget);
|
||||
A --> E(Progress Bar);
|
||||
A --> F(Start Button);
|
||||
A -- Contains --> SB(Status Bar);
|
||||
A --> CB(Cancel Button);
|
||||
A --> MB(Manage Presets Button);
|
||||
|
||||
B -- fileDroppedSignal --> X(Handle Input Files);
|
||||
C -- currentIndexChangedSignal --> Y(Update Preview);
|
||||
X -- Adds paths --> D;
|
||||
X -- Triggers --> Y;
|
||||
|
||||
Y -- Reads --> C;
|
||||
Y -- Uses --> J(Prediction Logic);
|
||||
Y -- Updates --> D;
|
||||
|
||||
F -- clickedSignal --> Z(Start Processing);
|
||||
CB -- clickedSignal --> AA(Cancel Processing);
|
||||
MB -- clickedSignal --> L(Preset Editor Dialog);
|
||||
|
||||
Z -- Starts --> BB(Processing Thread: QThread);
|
||||
AA -- Signals --> BB;
|
||||
|
||||
BB -- progressSignal(curr, total) --> E;
|
||||
BB -- fileStatusSignal(path, status, msg) --> D;
|
||||
BB -- finishedSignal(stats) --> AB(Handle Processing Finished);
|
||||
AB -- Updates --> SB;
|
||||
AB -- Enables/Disables --> F;
|
||||
AB -- Enables/Disables --> CB;
|
||||
AB -- Enables/Disables --> C;
|
||||
AB -- Enables/Disables --> B;
|
||||
|
||||
L -- Modifies --> H(Presets Dir: presets/*.json);
|
||||
end
|
||||
|
||||
subgraph Backend Logic (Existing + Refactored)
|
||||
H -- Loaded by --> C;
|
||||
H -- Loaded/Saved by --> L;
|
||||
|
||||
J -- Reads --> M(configuration.py / asset_processor.py);
|
||||
BB -- Runs --> K(Adapted main.py Logic);
|
||||
K -- Uses --> N(ProcessPoolExecutor);
|
||||
N -- Runs --> O(process_single_asset_wrapper);
|
||||
O -- Uses --> M;
|
||||
O -- Reports Status --> K;
|
||||
K -- Reports Progress --> BB;
|
||||
end
|
||||
|
||||
classDef gui fill:#f9f,stroke:#333,stroke-width:2px;
|
||||
classDef backend fill:#ccf,stroke:#333,stroke-width:2px;
|
||||
classDef thread fill:#ffc,stroke:#333,stroke-width:1px;
|
||||
class A,B,C,D,E,F,CB,MB,L,SB,X,Y,Z,AA,AB gui;
|
||||
class H,J,K,M,N,O backend;
|
||||
class BB thread;
|
||||
```
|
||||
|
||||
This plan provides a roadmap for the GUI development.
|
||||
@ -1,42 +0,0 @@
|
||||
# GUI Preset Selection Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Modify the GUI so that no preset is selected by default, and the preview table is only populated after the user explicitly selects a preset. This aims to prevent accidental processing with an unintended preset and clearly indicate to the user that a preset selection is required for preview.
|
||||
|
||||
## Plan
|
||||
|
||||
1. **Modify `gui/main_window.py`:**
|
||||
* Remove the logic that selects a default preset during initialization.
|
||||
* Initialize the preview table to display the text "please select a preset".
|
||||
* Disable the mechanism that triggers the `PredictionHandler` for preview generation until a preset is selected.
|
||||
* Update the slot connected to the preset selection signal:
|
||||
* When a preset is selected, clear the placeholder text and enable the preview generation mechanism.
|
||||
* Pass the selected preset configuration to the `PredictionHandler`.
|
||||
* Trigger the `PredictionHandler` to generate and display the preview.
|
||||
* (Optional but recommended) Add logic to handle the deselection of a preset, which should clear the preview table and display the "please select a preset" text again, and disable preview generation.
|
||||
|
||||
2. **Review `gui/prediction_handler.py`:**
|
||||
* Verify that the `PredictionHandler`'s methods that generate predictions (`get_detailed_file_predictions`) correctly handle being called only when a valid preset is provided. No major changes are expected here, but it's good practice to confirm.
|
||||
|
||||
3. **Update Preview Table Handling (`gui/preview_table_model.py` and `gui/main_window.py`):**
|
||||
* Ensure the `PreviewTableModel` can gracefully handle having no data when no preset is selected.
|
||||
* In `gui/main_window.py`, configure the `QTableView` or its parent widget to display the placeholder text "please select a preset" when the model is empty or no preset is active.
|
||||
|
||||
## Data Flow Change
|
||||
|
||||
The current data flow for preview generation is roughly:
|
||||
Initialization -> Default Preset Loaded -> Trigger PredictionHandler -> Update Preview Table
|
||||
|
||||
The proposed data flow would be:
|
||||
Initialization -> No Preset Selected -> Preview Table Empty/Placeholder -> User Selects Preset -> Trigger PredictionHandler with Selected Preset -> Update Preview Table
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[GUI Initialization] --> B{Is a Preset Selected?};
|
||||
B -- Yes (Current) --> C[Load Default Preset];
|
||||
B -- No (Proposed) --> D[Preview Table Empty/Placeholder];
|
||||
C --> E[Trigger PredictionHandler];
|
||||
D --> F[User Selects Preset];
|
||||
F --> E;
|
||||
E --> G[Update Preview Table];
|
||||
@ -1,59 +0,0 @@
|
||||
# Plan: Implement Alternating Row Colors Per Asset Group in GUI Preview Table
|
||||
|
||||
## Objective
|
||||
|
||||
Modify the GUI preview table to display alternating background colors for rows based on the asset group they belong to, rather than alternating colors for each individual row. The visual appearance should be similar to the default alternating row colors (dark greys, no border, no rounded corners).
|
||||
|
||||
## Current State
|
||||
|
||||
The preview table in the GUI uses a `QTableView` with `setAlternatingRowColors(True)` enabled, which applies alternating background colors based on the row index. The `PreviewTableModel` groups file prediction data by `source_asset` in its internal `_table_rows` structure and provides data to the view.
|
||||
|
||||
## Proposed Plan
|
||||
|
||||
To achieve alternating colors per asset group, we will implement custom coloring logic within the `PreviewTableModel`.
|
||||
|
||||
1. **Disable Default Alternating Colors:**
|
||||
* In `gui/main_window.py`, locate the initialization of the `preview_table_view` (a `QTableView`).
|
||||
* Change `self.preview_table_view.setAlternatingRowColors(True)` to `self.preview_table_view.setAlternatingRowColors(False)`.
|
||||
|
||||
2. **Modify `PreviewTableModel.data()`:**
|
||||
* Open `gui/preview_table_model.py`.
|
||||
* In the `data()` method, add a case to handle the `Qt.ItemDataRole.BackgroundRole`.
|
||||
* Inside this case, retrieve the `source_asset` for the current row from the `self._table_rows` structure.
|
||||
* Maintain a sorted list of unique `source_asset` values. This can be done when the data is set in `set_data()`.
|
||||
* Find the index of the current row's `source_asset` within this sorted list.
|
||||
* Based on whether the index is even or odd, return a `QColor` object representing one of the two desired grey colors.
|
||||
* Ensure that the `Qt.ItemDataRole.BackgroundRole` is handled correctly for all columns in the row.
|
||||
|
||||
3. **Define Colors:**
|
||||
* Define two `QColor` objects within the `PreviewTableModel` class to represent the two grey colors for alternating groups. These should be chosen to be visually similar to the default alternating row colors.
|
||||
|
||||
## Visual Representation of Data Flow with Custom Coloring
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[QTableView] --> B{Requests Data for Row/Column};
|
||||
B --> C[PreviewSortFilterProxyModel];
|
||||
C --> D[PreviewTableModel];
|
||||
D -- data(index, role) --> E{Check Role};
|
||||
E -- Qt.ItemDataRole.BackgroundRole --> F{Get source_asset for row};
|
||||
F --> G{Determine Asset Group Index};
|
||||
G --> H{Assign Color based on Index Parity};
|
||||
H --> I[Return QColor];
|
||||
E -- Other Roles --> J[Return Display/Tooltip/Foreground Data};
|
||||
I --> C;
|
||||
J --> C;
|
||||
C --> A{Displays Data with Custom Background Color};
|
||||
|
||||
style D fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style H fill:#ccf,stroke:#333,stroke-width:1px
|
||||
style I fill:#ccf,stroke:#333,stroke-width:1px
|
||||
```
|
||||
|
||||
## Implementation Steps (for Code Mode)
|
||||
|
||||
1. Modify `gui/main_window.py` to disable default alternating row colors.
|
||||
2. Modify `gui/preview_table_model.py` to:
|
||||
* Define the two grey `QColor` objects.
|
||||
* Update `set_data()` to create and store a sorted list of unique asset groups.
|
||||
* Implement the `Qt.ItemDataRole.BackgroundRole` logic in the `data()` method to return alternating colors based on the asset group index.
|
||||
@ -1,49 +0,0 @@
|
||||
# Plan: Enhance GUI Preview Table Coloring
|
||||
|
||||
## Objective
|
||||
|
||||
Modify the GUI preview table to apply status-based text coloring to all relevant cells in a row, providing a more consistent visual indication of a file's status.
|
||||
|
||||
## Current State
|
||||
|
||||
The `PreviewTableModel` in `gui/preview_table_model.py` currently applies status-based text colors only to the "Status" column (based on the main file's status) and the "Additional Files" column (based on the additional file's status). Other cells in the row do not have status-based coloring.
|
||||
|
||||
## Proposed Change
|
||||
|
||||
Extend the status-based text coloring logic in the `PreviewTableModel`'s `data()` method to apply the relevant status color to any cell that corresponds to either the main file or an additional file in that row.
|
||||
|
||||
## Plan
|
||||
|
||||
1. **Modify the `data()` method in `gui/preview_table_model.py`:**
|
||||
* Locate the section handling the `Qt.ItemDataRole.ForegroundRole`.
|
||||
* Currently, this section checks the column index (`col`) to decide which file's status to use for coloring (main file for `COL_STATUS`, additional file for `COL_ADDITIONAL_FILES`).
|
||||
* We will change this logic to determine which file (main or additional) the *current row and column* corresponds to, and then use that file's status to look up the color.
|
||||
* For columns related to the main file (`COL_STATUS`, `COL_PREDICTED_ASSET`, `COL_ORIGINAL_PATH`, `COL_PREDICTED_OUTPUT`, `COL_DETAILS`), if the row contains a `main_file`, use the `main_file`'s status for coloring.
|
||||
* For the `COL_ADDITIONAL_FILES` column, if the row contains `additional_file_details`, use the `additional_file_details`' status for coloring.
|
||||
* If a cell does not correspond to a file (e.g., a main file column in a row that only has an additional file), return `None` for the `ForegroundRole` to use the default text color.
|
||||
|
||||
## Detailed Steps
|
||||
|
||||
1. Open `gui/preview_table_model.py`.
|
||||
2. Navigate to the `data()` method.
|
||||
3. Find the `if role == Qt.ItemDataRole.ForegroundRole:` block.
|
||||
4. Inside this block, modify the logic to determine the `status` variable based on the current `col` and the presence of `main_file` or `additional_file_details` in `row_data`.
|
||||
5. Use the determined `status` to look up the color in `self.STATUS_COLORS`.
|
||||
6. Return the color if found, otherwise return `None`.
|
||||
|
||||
## Modified Color Logic Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[data(index, role)] --> B{role == Qt.ItemDataRole.ForegroundRole?};
|
||||
B -- Yes --> C{Determine relevant file for cell (row, col)};
|
||||
C -- Cell corresponds to Main File --> D{Get main_file status};
|
||||
C -- Cell corresponds to Additional File --> E{Get additional_file_details status};
|
||||
C -- Cell is empty --> F[status = None];
|
||||
D --> G{Lookup color in STATUS_COLORS};
|
||||
E --> G;
|
||||
F --> H[Return None];
|
||||
G -- Color found --> I[Return Color];
|
||||
G -- No color found --> H;
|
||||
B -- No --> J[Handle other roles];
|
||||
J --> K[Return data based on role];
|
||||
@ -1,90 +0,0 @@
|
||||
# GUI Preview Table Restructure Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Restructure the Graphical User Interface (GUI) preview table to group files by source asset and display "Ignored" and "Extra" files in a new "Additional Files" column, aligned with the mapped files of the same asset.
|
||||
|
||||
## Analysis
|
||||
|
||||
Based on the review of `gui/prediction_handler.py` and `gui/preview_table_model.py`:
|
||||
|
||||
* The `PredictionHandler` provides a flat list of file prediction dictionaries.
|
||||
* The `PreviewTableModel` currently stores and displays this flat list directly.
|
||||
* The `PreviewSortFilterProxyModel` sorts this flat list.
|
||||
* The data transformation to achieve the desired grouped layout must occur within the `PreviewTableModel`.
|
||||
|
||||
## Proposed Plan
|
||||
|
||||
1. **Modify `gui/preview_table_model.py`:**
|
||||
* **Add New Column:**
|
||||
* Define a new constant: `COL_ADDITIONAL_FILES = 5`.
|
||||
* Add "Additional Files" to the `_headers_detailed` list.
|
||||
* **Introduce New Internal Data Structure:**
|
||||
* Create a new internal list, `self._table_rows`, to store dictionaries representing the final rows to be displayed in the table.
|
||||
* **Update `set_data(self, data: list)`:**
|
||||
* Process the incoming flat `data` list (received from `PredictionHandler`).
|
||||
* Group file dictionaries by their `source_asset`.
|
||||
* Within each asset group, separate files into two lists:
|
||||
* `main_files`: Files with status "Mapped", "Model", or "Error".
|
||||
* `additional_files`: Files with status "Ignored", "Extra", "Unrecognised", or "Unmatched Extra".
|
||||
* Determine the maximum number of rows needed for this asset block: `max(len(main_files), len(additional_files))`.
|
||||
* Build the row dictionaries for `self._table_rows` for this asset block:
|
||||
* For `i` from 0 to `max_rows - 1`:
|
||||
* Get the `i`-th file from `main_files` (or `None` if `i` is out of bounds).
|
||||
* Get the `i`-th file from `additional_files` (or `None` if `i` is out of bounds).
|
||||
* Create a row dictionary containing:
|
||||
* `source_asset`: The asset name.
|
||||
* `predicted_asset`: From the `main_file` (if exists).
|
||||
* `details`: From the `main_file` (if exists).
|
||||
* `original_path`: From the `main_file` (if exists).
|
||||
* `additional_file_path`: Path from the `additional_file` (if exists).
|
||||
* `additional_file_details`: The original dictionary of the `additional_file` (if exists, for tooltips).
|
||||
* `is_main_row`: Boolean flag (True if this row corresponds to a file in `main_files`, False otherwise).
|
||||
* Append these row dictionaries to `self._table_rows`.
|
||||
* After processing all assets, call `self.beginResetModel()` and `self.endResetModel()`.
|
||||
* **Update `rowCount`:** Return `len(self._table_rows)` when in detailed mode.
|
||||
* **Update `columnCount`:** Return `len(self._headers_detailed)`.
|
||||
* **Update `data(self, index, role)`:**
|
||||
* Retrieve the row dictionary: `row_data = self._table_rows[index.row()]`.
|
||||
* For `Qt.ItemDataRole.DisplayRole`:
|
||||
* If `index.column()` is `COL_ADDITIONAL_FILES`, return `row_data.get('additional_file_path', '')`.
|
||||
* For other columns (`COL_STATUS`, `COL_PREDICTED_ASSET`, `COL_ORIGINAL_PATH`, `COL_DETAILS`), return data from the `main_file` part of `row_data` if `row_data['is_main_row']` is True, otherwise return an empty string or appropriate placeholder.
|
||||
* For `Qt.ItemDataRole.ToolTipRole`:
|
||||
* If `index.column()` is `COL_ADDITIONAL_FILES` and `row_data.get('additional_file_details')` exists, generate a tooltip using the status and details from `additional_file_details`.
|
||||
* For other columns, use the existing tooltip logic based on the `main_file` data.
|
||||
* For `Qt.ItemDataRole.ForegroundRole`:
|
||||
* Apply existing color-coding based on the status of the `main_file` if `row_data['is_main_row']` is True.
|
||||
* For the `COL_ADDITIONAL_FILES` cell and for rows where `row_data['is_main_row']` is False, use neutral styling (default text color).
|
||||
* **Update `headerData`:** Return the correct header for `COL_ADDITIONAL_FILES`.
|
||||
|
||||
2. **Modify `gui/preview_table_model.py` (`PreviewSortFilterProxyModel`):**
|
||||
* **Update `lessThan(self, left, right)`:**
|
||||
* Retrieve the row dictionaries for `left` and `right` indices from the source model (`model._table_rows[left.row()]`, etc.).
|
||||
* **Level 1: Source Asset:** Compare `source_asset` from the row dictionaries.
|
||||
* **Level 2: Row Type:** If assets are the same, compare `is_main_row` (True sorts before False).
|
||||
* **Level 3 (Main Rows):** If both are main rows (`is_main_row` is True), compare `original_path`.
|
||||
* **Level 4 (Additional-Only Rows):** If both are additional-only rows (`is_main_row` is False), compare `additional_file_path`.
|
||||
|
||||
## Clarifications & Decisions
|
||||
|
||||
* **Error Handling:** "Error" files will remain in the main columns, similar to "Mapped" files, with their current "Error" status.
|
||||
* **Sorting within Asset:** The proposed sorting logic within an asset block is acceptable (mapped rows by original path, additional-only rows by additional file path).
|
||||
* **Styling of Additional Column:** Use neutral text and background styling for the "Additional Files" column, relying on tooltips for specific file details.
|
||||
|
||||
## Mermaid Diagram (Updated Data Flow)
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[PredictionHandler] -- prediction_results_ready(flat_list) --> B(PreviewTableModel);
|
||||
subgraph PreviewTableModel
|
||||
C[set_data] -- Processes flat_list --> D{Internal Grouping & Transformation};
|
||||
D -- Creates --> E[_table_rows (Structured List)];
|
||||
F[data()] -- Reads from --> E;
|
||||
end
|
||||
B -- Provides data via data() --> G(QTableView via Proxy);
|
||||
|
||||
style B fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style C fill:#ccf,stroke:#333,stroke-width:1px
|
||||
style D fill:#lightgrey,stroke:#333,stroke-width:1px
|
||||
style E fill:#ccf,stroke:#333,stroke-width:1px
|
||||
style F fill:#ccf,stroke:#333,stroke-width:1px
|
||||
@ -1,123 +0,0 @@
|
||||
# Asset Processor GUI Refactor Plan
|
||||
|
||||
This document outlines the plan to refactor the Asset Processor GUI based on user requirements.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Improve File Visibility:** Display all files found within an asset in the preview list, including those that don't match the preset, are moved to 'Extra', or have errors, along with their status.
|
||||
2. **Integrate Preset Editor:** Move the preset editing functionality from the separate dialog into a collapsible panel within the main window.
|
||||
|
||||
## Goal 1: Improve File Visibility in Preview List
|
||||
|
||||
**Problem:** The current preview (`PredictionHandler` calling `AssetProcessor.predict_output_structure`) only shows files that successfully match a map type rule and get a predicted output name. It doesn't show files that are ignored, moved to 'Extra', or encounter errors during classification.
|
||||
|
||||
**Solution:** Leverage the more comprehensive classification logic already present in `AssetProcessor._inventory_and_classify_files` for the GUI preview.
|
||||
|
||||
**Plan Steps:**
|
||||
|
||||
1. **Modify `asset_processor.py`:**
|
||||
* Create a new method in `AssetProcessor`, perhaps named `get_detailed_file_predictions()`.
|
||||
* This new method will perform the core steps of `_setup_workspace()`, `_extract_input()`, and `_inventory_and_classify_files()`.
|
||||
* It will then iterate through *all* categories in `self.classified_files` ('maps', 'models', 'extra', 'ignored').
|
||||
* For each file, it will determine a 'status' (e.g., "Mapped", "Model", "Extra", "Ignored", "Error") and attempt to predict the output name (similar to `predict_output_structure` for maps, maybe just the original name for others).
|
||||
* It will return a more detailed list of dictionaries, each containing: `{'original_path': str, 'predicted_name': str | None, 'status': str, 'details': str | None}`.
|
||||
* Crucially, this method will *not* perform the actual processing (`_process_maps`, `_merge_maps`, etc.) or file moving, only the classification and prediction. It should also include cleanup (`_cleanup_workspace`).
|
||||
2. **Modify `gui/prediction_handler.py`:**
|
||||
* Update `PredictionHandler.run_prediction` to call the new `AssetProcessor.get_detailed_file_predictions()` method instead of `predict_output_structure()`.
|
||||
* Adapt the code that processes the results to handle the new dictionary format (including the 'status' and 'details' fields).
|
||||
* Emit this enhanced list via the `prediction_results_ready` signal.
|
||||
3. **Modify `gui/main_window.py`:**
|
||||
* In `setup_ui`, add a new column to `self.preview_table` for "Status". Adjust column count and header labels.
|
||||
* In `on_prediction_results_ready`, populate the new "Status" column using the data received from `PredictionHandler`.
|
||||
* Consider adding tooltips to the status column to show the 'details' (e.g., the reason for being ignored or moved to extra).
|
||||
* Optionally, use background colors or icons in the status column for better visual distinction.
|
||||
|
||||
## Goal 2: Integrate Preset Editor into Main Window
|
||||
|
||||
**Problem:** Preset editing requires opening a separate modal dialog, interrupting the main workflow.
|
||||
|
||||
**Solution:** Embed the preset editing controls directly into the main window within a collapsible panel.
|
||||
|
||||
**Plan Steps:**
|
||||
|
||||
1. **Modify `gui/main_window.py` - UI Changes:**
|
||||
* Remove the "Manage Presets" button (`self.manage_presets_button`).
|
||||
* Add a collapsible panel (potentially using a `QFrame` with show/hide logic triggered by a button, or a `QDockWidget` if more appropriate) to the left side of the main layout.
|
||||
* Inside this panel:
|
||||
* Add the `QListWidget` for displaying presets (`self.preset_list`).
|
||||
* Add the "New" and "Delete" buttons below the list. (The "Load" button becomes implicit - selecting a preset in the list loads it into the editor).
|
||||
* Recreate the `QTabWidget` (`self.preset_editor_tabs`) with the "General & Naming" and "Mapping & Rules" tabs.
|
||||
* Recreate *all* the widgets currently inside the `PresetEditorDialog` tabs (QLineEdit, QTextEdit, QSpinBox, QListWidget+controls, QTableWidget+controls) within the corresponding tabs in the main window's panel. Give them appropriate instance names (e.g., `self.editor_preset_name`, `self.editor_supplier_name`, etc.).
|
||||
* Add "Save" and "Save As..." buttons within the collapsible panel, likely at the bottom.
|
||||
2. **Modify `gui/main_window.py` - Logic Integration:**
|
||||
* Adapt the `populate_presets` method to populate the new `self.preset_list` in the panel.
|
||||
* Connect `self.preset_list.currentItemChanged` to a new method `load_selected_preset_for_editing`. This method will handle checking for unsaved changes in the editor panel and then load the selected preset's data into the editor widgets (similar to `PresetEditorDialog.load_preset`).
|
||||
* Implement `save_preset`, `save_preset_as`, `new_preset`, `delete_preset` methods directly within `MainWindow`, adapting the logic from `PresetEditorDialog`. These will interact with the editor widgets in the panel.
|
||||
* Implement `check_unsaved_changes` logic for the editor panel, prompting the user if they try to load/create/delete a preset or close the application with unsaved edits in the panel.
|
||||
* Connect the editor widgets' change signals (`textChanged`, `valueChanged`, `itemChanged`, etc.) to a `mark_editor_unsaved` method in `MainWindow`.
|
||||
* Ensure the main preset selection `QComboBox` (`self.preset_combo`) is repopulated when presets are saved/deleted via the editor panel.
|
||||
3. **Cleanup:**
|
||||
* Delete the `gui/preset_editor_dialog.py` file.
|
||||
* Remove imports and references to `PresetEditorDialog` from `gui/main_window.py`.
|
||||
|
||||
## Visual Plan
|
||||
|
||||
**Current Layout:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Current Main Window Layout"
|
||||
A[Preset Combo + Manage Button] --> B(Drag & Drop Area);
|
||||
B --> C{File Preview Table};
|
||||
C --> D[Progress Bar];
|
||||
D --> E[Options + Start/Cancel Buttons];
|
||||
end
|
||||
|
||||
subgraph "Current Preset Editor (Separate Dialog)"
|
||||
F[Preset List + Load/New/Delete] --> G{Tab Widget};
|
||||
subgraph "Tab Widget"
|
||||
G1[General & Naming Tab]
|
||||
G2[Mapping & Rules Tab]
|
||||
end
|
||||
G --> H[Save / Save As / Close Buttons];
|
||||
end
|
||||
|
||||
A -- Manage Button Click --> F;
|
||||
|
||||
style F fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style G fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style H fill:#f9f,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
**Proposed Layout:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Proposed Main Window Layout"
|
||||
direction LR
|
||||
subgraph "Collapsible Preset Editor Panel (Left)"
|
||||
P_List[Preset List] --> P_Buttons[New / Delete Buttons]
|
||||
P_Buttons --> P_Tabs{Tab Widget}
|
||||
subgraph "Editor Tabs"
|
||||
P_Tab1[General & Naming]
|
||||
P_Tab2[Mapping & Rules]
|
||||
end
|
||||
P_Tabs --> P_Save[Save / Save As Buttons]
|
||||
end
|
||||
|
||||
subgraph "Main Area (Right)"
|
||||
M_Preset[Preset Combo (for processing)] --> M_DragDrop(Drag & Drop Area)
|
||||
M_DragDrop --> M_Preview{File Preview Table (with Status Column)}
|
||||
M_Preview --> M_Progress[Progress Bar]
|
||||
M_Progress --> M_Controls[Options + Start/Cancel Buttons]
|
||||
end
|
||||
|
||||
P_List -- Selection Loads --> P_Tabs;
|
||||
P_Save -- Updates --> P_List;
|
||||
P_List -- Updates --> M_Preset;
|
||||
|
||||
style M_Preview fill:#ccf,stroke:#333,stroke-width:2px
|
||||
style P_List fill:#cfc,stroke:#333,stroke-width:2px
|
||||
style P_Tabs fill:#cfc,stroke:#333,stroke-width:2px
|
||||
style P_Save fill:#cfc,stroke:#333,stroke-width:2px
|
||||
end
|
||||
@ -1,63 +0,0 @@
|
||||
# Plan: Update GUI Preview Status
|
||||
|
||||
**Objective:** Modify the Asset Processor GUI preview to distinguish between files explicitly marked as "Extra" by preset patterns and those that are simply unclassified.
|
||||
|
||||
**Current Statuses:**
|
||||
* Mapped
|
||||
* Ignored
|
||||
* Extra (Includes both explicitly matched and unclassified files)
|
||||
|
||||
**Proposed Statuses:**
|
||||
* Mapped
|
||||
* Ignored
|
||||
* Extra (Files explicitly matched by `move_to_extra_patterns` in the preset)
|
||||
* Unrecognised (Files not matching any map, model, or explicit extra pattern)
|
||||
|
||||
**Visual Plan:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start: User Request] --> B{Analyze Request: Split 'Extra' status};
|
||||
B --> C{Info Gathering};
|
||||
C --> D[Read gui/prediction_handler.py];
|
||||
D --> E[Read asset_processor.py];
|
||||
E --> F[Read Presets/Poliigon.json];
|
||||
F --> G[Read gui/main_window.py];
|
||||
G --> H{Identify Code Locations};
|
||||
H --> I[asset_processor.py: get_detailed_file_predictions()];
|
||||
H --> J[gui/main_window.py: on_prediction_results_ready()];
|
||||
I --> K{Plan Code Changes};
|
||||
J --> K;
|
||||
K --> L[Modify asset_processor.py: Differentiate status based on 'reason'];
|
||||
K --> M[Modify gui/main_window.py: Add color rule for 'Unrecognised' (#92371f)];
|
||||
L --> N{Final Plan};
|
||||
M --> N;
|
||||
N --> O[Present Plan to User];
|
||||
O --> P{User Approval + Color Choice};
|
||||
P --> Q[Switch to Code Mode for Implementation];
|
||||
|
||||
subgraph "Code Modification"
|
||||
L
|
||||
M
|
||||
end
|
||||
|
||||
subgraph "Information Gathering"
|
||||
D
|
||||
E
|
||||
F
|
||||
G
|
||||
end
|
||||
```
|
||||
|
||||
**Implementation Steps:**
|
||||
|
||||
1. **Modify `asset_processor.py` (`get_detailed_file_predictions` method):**
|
||||
* Locate the loop processing the `self.classified_files["extra"]` list (around line 1448).
|
||||
* Inside this loop, check the `reason` associated with each file:
|
||||
* If `reason == 'Unclassified'`, set the output `status` to `"Unrecognised"`.
|
||||
* Otherwise (if the reason indicates an explicit pattern match), set the output `status` to `"Extra"`.
|
||||
* Adjust the `details` string provided in the output for clarity (e.g., show pattern match reason for "Extra", maybe just "[Unrecognised]" for the new status).
|
||||
|
||||
2. **Modify `gui/main_window.py` (`on_prediction_results_ready` method):**
|
||||
* Locate the section where text color is applied based on the `status` (around line 673).
|
||||
* Add a new `elif` condition to handle `status == "Unrecognised"` and assign it the color `QColor("#92371f")`.
|
||||
@ -1,34 +0,0 @@
|
||||
# Plan to Resolve ISSUE-011: Blender nodegroup script creates empty assets for skipped items
|
||||
|
||||
**Issue:** The Blender nodegroup creation script (`blenderscripts/create_nodegroups.py`) creates empty asset entries in the target .blend file for assets belonging to categories that the script is designed to skip, even though it correctly identifies them as skippable.
|
||||
|
||||
**Root Cause:** The script creates the parent node group, marks it as an asset, and applies tags *before* checking if the asset category is one that should be skipped for full nodegroup generation.
|
||||
|
||||
**Plan:**
|
||||
|
||||
1. **Analyze `blenderscripts/create_nodegroups.py` (Completed):** Confirmed that parent group creation and asset marking occur before the asset category skip check.
|
||||
2. **Modify `blenderscripts/create_nodegroups.py`:**
|
||||
* Relocate the code block responsible for creating/updating the parent node group, marking it as an asset, and applying tags (currently lines 605-645) to *after* the conditional check `if asset_category not in CATEGORIES_FOR_NODEGROUP_GENERATION:` (line 646).
|
||||
* This ensures that if an asset's category is in the list of categories to be skipped, the `continue` statement will be hit before any actions are taken to create the parent asset entry in the Blender file.
|
||||
|
||||
3. **Testing:**
|
||||
* Use test assets that represent both categories that *should* and *should not* result in full nodegroup generation based on the `CATEGORIES_FOR_NODEGROUP_GENERATION` list.
|
||||
* Run the asset processor with these test assets, ensuring the Blender script is executed.
|
||||
* Inspect the resulting `.blend` file to confirm:
|
||||
* No `PBRSET_` node groups are created for assets belonging to skipped categories.
|
||||
* `PBRSET_` node groups are correctly created and populated for assets belonging to categories in `CATEGORIES_FOR_NODEGROUP_GENERATION`.
|
||||
|
||||
4. **Update Ticket Status:**
|
||||
* Once the fix is implemented and verified, update the `Status` field in `Tickets/ISSUE-011-blender-nodegroup-empty-assets.md` to `Resolved`.
|
||||
|
||||
**Logic Flow:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[create_nodegroups.py] --> B{Load Asset Metadata};
|
||||
B --> C{Determine Asset Category};
|
||||
C --> D{Is Category Skipped?};
|
||||
D -- Yes --> E[Exit Processing for Asset];
|
||||
D -- No --> F{Create/Update Parent Group};
|
||||
F --> G{Mark as Asset & Add Tags};
|
||||
G --> H{Proceed with Child Group Creation etc.};
|
||||
@ -1,68 +0,0 @@
|
||||
# Map Variant Handling Plan (Revised)
|
||||
|
||||
**Goal:**
|
||||
|
||||
1. Ensure map types listed in a new `RESPECT_VARIANT_MAP_TYPES` config setting (initially just "COL") always receive a numeric suffix (`-1`, `-2`, etc.), based on their order determined by preset keywords and alphabetical sorting within keywords.
|
||||
2. Ensure all other map types *never* receive a numeric suffix.
|
||||
3. Correctly prioritize 16-bit map variants (identified by `bit_depth_variants` in presets) over their 8-bit counterparts, ensuring the 8-bit version is ignored/marked as extra and the 16-bit version is correctly classified ("Mapped") in the GUI preview.
|
||||
|
||||
**Affected Files:**
|
||||
|
||||
* `config.py`: To define the `RESPECT_VARIANT_MAP_TYPES` list.
|
||||
* `asset_processor.py`: To modify the classification and suffix assignment logic according to the new rule.
|
||||
* `Presets/Poliigon.json`: To remove the conflicting pattern.
|
||||
|
||||
**Plan Details:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start] --> B(Modify config.py);
|
||||
B --> C(Modify asset_processor.py);
|
||||
C --> D(Modify Presets/Poliigon.json);
|
||||
D --> E{Review Revised Plan};
|
||||
E -- Approve --> F(Optional: Write Plan to MD);
|
||||
F --> G(Switch to Code Mode);
|
||||
E -- Request Changes --> B;
|
||||
G --> H[End Plan];
|
||||
|
||||
subgraph Modifications
|
||||
B[1. Add RESPECT_VARIANT_MAP_TYPES list to config.py]
|
||||
C[2. Update suffix logic in asset_processor.py (_inventory_and_classify_files)]
|
||||
D[3. Remove "*_16BIT*" pattern from move_to_extra_patterns in Presets/Poliigon.json]
|
||||
end
|
||||
```
|
||||
|
||||
1. **Modify `config.py`:**
|
||||
* **Action:** Introduce a new configuration list named `RESPECT_VARIANT_MAP_TYPES`.
|
||||
* **Value:** Initialize it as `RESPECT_VARIANT_MAP_TYPES = ["COL"]`.
|
||||
* **Location:** Add this near other map-related settings like `STANDARD_MAP_TYPES`.
|
||||
* **Purpose:** To explicitly define which map types should always respect variant numbering via suffixes.
|
||||
|
||||
2. **Modify `asset_processor.py`:**
|
||||
* **File:** `asset_processor.py`
|
||||
* **Method:** `_inventory_and_classify_files`
|
||||
* **Location:** Within Step 5, replacing the suffix assignment logic (currently lines ~470-474).
|
||||
* **Action:** Implement the new conditional logic for assigning the `final_map_type`.
|
||||
* **New Logic:** Inside the loop iterating through `final_ordered_candidates` (for each `base_map_type`):
|
||||
```python
|
||||
# Determine final map type based on the new rule
|
||||
if base_map_type in self.config.respect_variant_map_types: # Check the new config list
|
||||
# Always assign suffix for types in the list
|
||||
final_map_type = f"{base_map_type}-{i + 1}"
|
||||
else:
|
||||
# Never assign suffix for types NOT in the list
|
||||
final_map_type = base_map_type
|
||||
|
||||
# Assign to the final map list entry
|
||||
final_map_list.append({
|
||||
"map_type": final_map_type,
|
||||
# ... rest of the dictionary assignment ...
|
||||
})
|
||||
```
|
||||
* **Purpose:** To implement the strict rule: only types in `RESPECT_VARIANT_MAP_TYPES` get a suffix; all others do not.
|
||||
|
||||
3. **Modify `Presets/Poliigon.json`:**
|
||||
* **File:** `Presets/Poliigon.json`
|
||||
* **Location:** Within the `move_to_extra_patterns` list (currently line ~28).
|
||||
* **Action:** Remove the string `"*_16BIT*"`.
|
||||
* **Purpose:** To prevent premature classification of 16-bit variants as "Extra", allowing the specific 16-bit prioritization logic to function correctly.
|
||||
@ -1,90 +0,0 @@
|
||||
# Memory Optimization Plan: Strategy 2 - Load Grayscale Directly
|
||||
|
||||
This plan outlines the steps to implement memory optimization strategy #2, which involves loading known grayscale map types directly as grayscale images using OpenCV's `IMREAD_GRAYSCALE` flag. This reduces the memory footprint compared to loading them with `IMREAD_UNCHANGED` and then potentially converting later.
|
||||
|
||||
## 1. Identify Target Grayscale Map Types
|
||||
|
||||
Define a list of map type names (case-insensitive check recommended during implementation) that should always be treated as single-channel grayscale data.
|
||||
|
||||
**Initial List:**
|
||||
```python
|
||||
GRAYSCALE_MAP_TYPES = ['HEIGHT', 'ROUGH', 'METAL', 'AO', 'OPC', 'MASK']
|
||||
```
|
||||
*(Note: This list might need adjustment based on specific preset configurations or workflow requirements.)*
|
||||
|
||||
## 2. Modify `_process_maps` Loading Logic
|
||||
|
||||
Locate the primary image loading section within the `_process_maps` method in `asset_processor.py` (around line 608).
|
||||
|
||||
**Change:** Before calling `cv2.imread`, determine the correct flag based on the `map_type`:
|
||||
|
||||
```python
|
||||
# (Define GRAYSCALE_MAP_TYPES list earlier in the scope or class)
|
||||
|
||||
# ... inside the loop ...
|
||||
full_source_path = self.temp_dir / source_path_rel
|
||||
|
||||
# Determine the read flag
|
||||
read_flag = cv2.IMREAD_GRAYSCALE if map_type.upper() in GRAYSCALE_MAP_TYPES else cv2.IMREAD_UNCHANGED
|
||||
log.debug(f"Loading source {source_path_rel.name} with flag: {'GRAYSCALE' if read_flag == cv2.IMREAD_GRAYSCALE else 'UNCHANGED'}")
|
||||
|
||||
# Load the image using the determined flag
|
||||
img_loaded = cv2.imread(str(full_source_path), read_flag)
|
||||
|
||||
if img_loaded is None:
|
||||
raise AssetProcessingError(f"Failed to load image file: {full_source_path.name} with flag {read_flag}")
|
||||
|
||||
# ... rest of the processing logic ...
|
||||
```
|
||||
|
||||
## 3. Modify `_merge_maps` Loading Logic
|
||||
|
||||
Locate the image loading section within the resolution loop in the `_merge_maps` method (around line 881).
|
||||
|
||||
**Change:** Apply the same conditional logic to determine the `imread` flag when loading input maps for merging:
|
||||
|
||||
```python
|
||||
# ... inside the loop ...
|
||||
input_file_path = self.temp_dir / res_details['path']
|
||||
|
||||
# Determine the read flag (reuse GRAYSCALE_MAP_TYPES list)
|
||||
read_flag = cv2.IMREAD_GRAYSCALE if map_type.upper() in GRAYSCALE_MAP_TYPES else cv2.IMREAD_UNCHANGED
|
||||
log.debug(f"Loading merge input {input_file_path.name} ({map_type}) with flag: {'GRAYSCALE' if read_flag == cv2.IMREAD_GRAYSCALE else 'UNCHANGED'}")
|
||||
|
||||
# Load the image using the determined flag
|
||||
img = cv2.imread(str(input_file_path), read_flag)
|
||||
|
||||
if img is None:
|
||||
raise AssetProcessingError(f"Failed to load merge input {input_file_path.name} with flag {read_flag}")
|
||||
|
||||
# ... rest of the merging logic ...
|
||||
```
|
||||
|
||||
## 4. Verification
|
||||
|
||||
During implementation in `code` mode:
|
||||
* Ensure the `GRAYSCALE_MAP_TYPES` list is defined appropriately (e.g., as a class constant or module-level constant).
|
||||
* Confirm that downstream code (e.g., stats calculation, channel extraction, data type conversions) correctly handles numpy arrays that might be 2D (grayscale) instead of 3D (BGR/BGRA). The existing code appears to handle this, but it's important to verify.
|
||||
|
||||
## Mermaid Diagram of Change
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph _process_maps
|
||||
A[Loop through map_info] --> B{Is map_type Grayscale?};
|
||||
B -- Yes --> C[imread(path, GRAYSCALE)];
|
||||
B -- No --> D[imread(path, UNCHANGED)];
|
||||
C --> E[Process Image];
|
||||
D --> E;
|
||||
end
|
||||
subgraph _merge_maps
|
||||
F[Loop through resolutions] --> G[Loop through required_input_types];
|
||||
G --> H{Is map_type Grayscale?};
|
||||
H -- Yes --> I[imread(path, GRAYSCALE)];
|
||||
H -- No --> J[imread(path, UNCHANGED)];
|
||||
I --> K[Use Image in Merge];
|
||||
J --> K;
|
||||
end
|
||||
|
||||
style B fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style H fill:#f9f,stroke:#333,stroke-width:2px
|
||||
@ -1,103 +0,0 @@
|
||||
# Plan: Implement Input-Based Output Format Logic
|
||||
|
||||
This plan outlines the steps to modify the Asset Processor Tool to determine the output format of texture maps based on the input file format and specific rules.
|
||||
|
||||
## Requirements Summary
|
||||
|
||||
Based on user clarifications:
|
||||
|
||||
1. **JPG Input -> JPG Output:** If the original source map is a JPG, the output for that map (at all processed resolutions) will also be JPG (8-bit).
|
||||
2. **TIF Input -> PNG/EXR Output:** If the original source map is a TIF, the output will be PNG (if the target bit depth is 8-bit, or if 16-bit PNG is the configured preference) or EXR (if the target bit depth is 16-bit and EXR is the configured preference).
|
||||
3. **Other Inputs (PNG, etc.) -> Configured Output:** For other input formats, the output will follow the existing logic based on target bit depth (using configured 16-bit or 8-bit formats, typically EXR/PNG).
|
||||
4. **`force_8bit` Rule:** If a map type has a `force_8bit` rule, it overrides the input format. Even if the input was 16-bit TIF, the output will be 8-bit PNG.
|
||||
5. **Merged Maps:** The output format is determined by the highest format in the hierarchy (EXR > TIF > PNG > JPG) based on the *original* formats of the input files used in the merge. However, if the highest format is TIF, the actual output will be PNG/EXR based on the target bit depth. The target bit depth itself is determined separately by the merge rule's `output_bit_depth` setting.
|
||||
6. **JPG Resizing:** Resized JPGs will be saved as JPG.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
**Phase 1: Data Gathering Enhancement**
|
||||
|
||||
1. **Modify `_inventory_and_classify_files` in `asset_processor.py`:**
|
||||
* When classifying map files, extract and store the original file extension (e.g., `.jpg`, `.tif`, `.png`) along with the `source_path`, `map_type`, etc., within the `self.classified_files["maps"]` list.
|
||||
|
||||
**Phase 2: Implement New Logic for Individual Maps (`_process_maps`)**
|
||||
|
||||
1. **Modify `_process_maps` in `asset_processor.py`:**
|
||||
* Inside the loop processing each `map_info`:
|
||||
* Retrieve the stored original file extension.
|
||||
* Determine the target output bit depth (8 or 16) using the existing logic (`config.get_bit_depth_rule`, source data type).
|
||||
* **Implement New Format Determination:**
|
||||
* Initialize `output_format` and `output_ext`.
|
||||
* Check the `force_8bit` rule first: If the rule is `force_8bit`, set `output_format = 'png'` and `output_ext = '.png'`, regardless of input format.
|
||||
* If not `force_8bit`:
|
||||
* If `original_extension == '.jpg'` and `target_bit_depth == 8`: Set `output_format = 'jpg'`, `output_ext = '.jpg'`.
|
||||
* If `original_extension == '.tif'`:
|
||||
* If `target_bit_depth == 16`: Determine format (EXR/PNG) and extension based on `config.get_16bit_output_formats()`.
|
||||
* If `target_bit_depth == 8`: Set `output_format = 'png'`, `output_ext = '.png'`.
|
||||
* If `original_extension` is neither `.jpg` nor `.tif` (e.g., `.png`):
|
||||
* If `target_bit_depth == 16`: Determine format (EXR/PNG) and extension based on `config.get_16bit_output_formats()`.
|
||||
* If `target_bit_depth == 8`: Set `output_format = config.get_8bit_output_format()` (likely 'png'), `output_ext = f".{output_format}"`.
|
||||
* **Remove Old Logic:** Delete the code block that checks `self.config.resolution_threshold_for_jpg`.
|
||||
* Set `save_params` based on the *newly determined* `output_format` (e.g., `cv2.IMWRITE_JPEG_QUALITY` for JPG, `cv2.IMWRITE_PNG_COMPRESSION` for PNG).
|
||||
* Proceed with data type conversion (if needed based on target bit depth and format requirements like EXR needing float16) and saving using `cv2.imwrite` with the determined `output_path_temp` (using the new `output_ext`) and `save_params`. Ensure fallback logic (e.g., EXR -> PNG) still functions correctly if needed.
|
||||
|
||||
**Phase 3: Implement New Logic for Merged Maps (`_merge_maps`)**
|
||||
|
||||
1. **Modify `_merge_maps` in `asset_processor.py`:**
|
||||
* Inside the loop for each `current_res_key`:
|
||||
* When loading input maps (`loaded_inputs`), also retrieve and store their *original* file extensions (obtained during classification and now available via `self.classified_files["maps"]`, potentially needing a lookup based on `res_details['path']` or storing it earlier).
|
||||
* **Determine Highest Input Format:** Iterate through the original extensions of the loaded inputs for this resolution. Use the hierarchy (EXR > TIF > PNG > JPG) to find the highest format present.
|
||||
* **Determine Final Output Format:**
|
||||
* Start with the `highest_input_format`.
|
||||
* If `highest_input_format == 'tif'`:
|
||||
* Check the target bit depth determined by the merge rule (`output_bit_depth`).
|
||||
* If `target_bit_depth == 16`: Set final format based on `config.get_16bit_output_formats()` (EXR/PNG).
|
||||
* If `target_bit_depth == 8`: Set final format to `png`.
|
||||
* Otherwise (JPG, PNG, EXR), the final format is the `highest_input_format`.
|
||||
* Set `output_ext` based on the `final_output_format`.
|
||||
* Set `save_params` based on the `final_output_format`.
|
||||
* Proceed with merging channels, converting the merged data to the target bit depth specified by the *merge rule*, and saving using `cv2.imwrite` with the determined `merged_output_path_temp` (using the new `output_ext`) and `save_params`.
|
||||
|
||||
**Phase 4: Configuration and Documentation**
|
||||
|
||||
1. **Modify `config.py`:**
|
||||
* Comment out or remove the `RESOLUTION_THRESHOLD_FOR_JPG` variable as it's no longer used. Add a comment explaining why it was removed.
|
||||
2. **Update `readme.md`:**
|
||||
* Modify the "Features" section (around line 21) and the "Configuration" section (around lines 36-40, 86) to accurately describe the new output format logic:
|
||||
* Explain that JPG inputs result in JPG outputs.
|
||||
* Explain that TIF inputs result in PNG/EXR outputs based on target bit depth and config.
|
||||
* Explain the merged map format determination based on input hierarchy (with the TIF->PNG/EXR adjustment).
|
||||
* Mention the removal of the JPG resolution threshold.
|
||||
|
||||
## Visual Plan (Mermaid)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start] --> B(Phase 1: Enhance Classification);
|
||||
B --> B1(Store original extension in classified_files['maps']);
|
||||
B1 --> C(Phase 2: Modify _process_maps);
|
||||
C --> C1(Get original extension);
|
||||
C --> C2(Determine target bit depth);
|
||||
C --> C3(Apply New Format Logic);
|
||||
C3 -- force_8bit --> C3a[Format=PNG];
|
||||
C3 -- input=.jpg, 8bit --> C3b[Format=JPG];
|
||||
C3 -- input=.tif, 16bit --> C3c[Format=EXR/PNG (Config)];
|
||||
C3 -- input=.tif, 8bit --> C3d[Format=PNG];
|
||||
C3 -- input=other, 16bit --> C3c;
|
||||
C3 -- input=other, 8bit --> C3e[Format=PNG (Config)];
|
||||
C3a --> C4(Remove JPG Threshold Check);
|
||||
C3b --> C4;
|
||||
C3c --> C4;
|
||||
C3d --> C4;
|
||||
C3e --> C4;
|
||||
C4 --> C5(Set Save Params & Save);
|
||||
C5 --> D(Phase 3: Modify _merge_maps);
|
||||
D --> D1(Get original extensions of inputs);
|
||||
D --> D2(Find highest format via hierarchy);
|
||||
D2 --> D3(Adjust TIF -> PNG/EXR based on target bit depth);
|
||||
D3 --> D4(Determine target bit depth from rule);
|
||||
D4 --> D5(Set Save Params & Save Merged);
|
||||
D5 --> E(Phase 4: Config & Docs);
|
||||
E --> E1(Update config.py - Remove threshold);
|
||||
E --> E2(Update readme.md);
|
||||
E2 --> F[End];
|
||||
@ -1,127 +0,0 @@
|
||||
# Revised Plan: Implement Input-Based Output Format Logic with JPG Threshold Override
|
||||
|
||||
This plan outlines the steps to modify the Asset Processor Tool to determine the output format of texture maps based on the input file format, specific rules, and a JPG resolution threshold override.
|
||||
|
||||
## Requirements Summary (Revised)
|
||||
|
||||
Based on user clarifications:
|
||||
|
||||
1. **JPG Threshold Override:** If the target output bit depth is 8-bit AND the image resolution is greater than or equal to `RESOLUTION_THRESHOLD_FOR_JPG` (defined in `config.py`), the output format **must** be JPG.
|
||||
2. **Input-Based Logic (if threshold not met):**
|
||||
* **JPG Input -> JPG Output:** If the original source map is JPG and the target is 8-bit (and below threshold), output JPG.
|
||||
* **TIF Input -> PNG/EXR Output:** If the original source map is TIF:
|
||||
* If target is 16-bit, output EXR or PNG based on `OUTPUT_FORMAT_16BIT_PRIMARY` config.
|
||||
* If target is 8-bit (and below threshold), output PNG.
|
||||
* **Other Inputs (PNG, etc.) -> Configured Output:** For other input formats (and below threshold if 8-bit):
|
||||
* If target is 16-bit, output EXR or PNG based on `OUTPUT_FORMAT_16BIT_PRIMARY` config.
|
||||
* If target is 8-bit, output PNG (or format specified by `OUTPUT_FORMAT_8BIT`).
|
||||
3. **`force_8bit` Rule:** If a map type has a `force_8bit` rule, the target bit depth is 8-bit. The output format will then be determined by the JPG threshold override or the input-based logic (resulting in JPG or PNG).
|
||||
4. **Merged Maps:**
|
||||
* Determine target `output_bit_depth` from the merge rule (`respect_inputs`, `force_8bit`, etc.).
|
||||
* **Check JPG Threshold Override:** If target `output_bit_depth` is 8-bit AND resolution >= threshold, the final output format is JPG.
|
||||
* **Else (Hierarchy Logic):** Determine the highest format among original inputs (EXR > TIF > PNG > JPG).
|
||||
* If highest was TIF, adjust based on target bit depth (16-bit -> EXR/PNG config; 8-bit -> PNG).
|
||||
* Otherwise, use the highest format found (EXR, PNG, JPG).
|
||||
* **JPG 8-bit Check:** If the final format is JPG but the target bit depth was 16, force the merged data to 8-bit before saving.
|
||||
5. **JPG Resizing:** Resized JPGs will be saved as JPG if the logic determines JPG as the output format.
|
||||
|
||||
## Implementation Plan (Revised)
|
||||
|
||||
**Phase 1: Data Gathering Enhancement** (Already Done & Correct)
|
||||
|
||||
* `_inventory_and_classify_files` stores the original file extension in `self.classified_files["maps"]`.
|
||||
|
||||
**Phase 2: Modify `_process_maps`**
|
||||
|
||||
1. Retrieve the `original_extension` from `map_info`.
|
||||
2. Determine the target `output_bit_depth` (8 or 16).
|
||||
3. Get the `threshold = self.config.resolution_threshold_for_jpg`.
|
||||
4. Get the `target_dim` for the current resolution loop iteration.
|
||||
5. **New Format Logic (Revised):**
|
||||
* Initialize `output_format`, `output_ext`, `save_params`, `needs_float16`.
|
||||
* **Check JPG Threshold Override:**
|
||||
* If `output_bit_depth == 8` AND `target_dim >= threshold`: Set format to JPG, set JPG params.
|
||||
* **Else (Apply Input/Rule-Based Logic):**
|
||||
* If `bit_depth_rule == 'force_8bit'`: Set format to PNG (8-bit), set PNG params.
|
||||
* Else if `original_extension == '.jpg'` and `output_bit_depth == 8`: Set format to JPG, set JPG params.
|
||||
* Else if `original_extension == '.tif'`:
|
||||
* If `output_bit_depth == 16`: Set format to EXR/PNG (16-bit config), set params, set `needs_float16` if EXR.
|
||||
* If `output_bit_depth == 8`: Set format to PNG, set PNG params.
|
||||
* Else (other inputs like `.png`):
|
||||
* If `output_bit_depth == 16`: Set format to EXR/PNG (16-bit config), set params, set `needs_float16` if EXR.
|
||||
* If `output_bit_depth == 8`: Set format to PNG (8-bit config), set PNG params.
|
||||
6. Proceed with data type conversion and saving.
|
||||
|
||||
**Phase 3: Modify `_merge_maps`**
|
||||
|
||||
1. Retrieve original extensions of inputs (`input_original_extensions`).
|
||||
2. Determine target `output_bit_depth` from the merge rule.
|
||||
3. Get the `threshold = self.config.resolution_threshold_for_jpg`.
|
||||
4. Get the `target_dim` for the current resolution loop iteration.
|
||||
5. **New Format Logic (Revised):**
|
||||
* Initialize `final_output_format`, `output_ext`, `save_params`, `needs_float16`.
|
||||
* **Check JPG Threshold Override:**
|
||||
* If `output_bit_depth == 8` AND `target_dim >= threshold`: Set `final_output_format = 'jpg'`.
|
||||
* **Else (Apply Hierarchy/Rule-Based Logic):**
|
||||
* Determine `highest_input_format` (EXR > TIF > PNG > JPG).
|
||||
* Start with `final_output_format = highest_input_format`.
|
||||
* If `highest_input_format == 'tif'`: Adjust based on target bit depth (16->EXR/PNG config; 8->PNG).
|
||||
* Set `output_format = final_output_format`.
|
||||
* Set `output_ext`, `save_params`, `needs_float16` based on `output_format`.
|
||||
* **JPG 8-bit Check:** If `output_format == 'jpg'` and `output_bit_depth == 16`, force final merged data to 8-bit before saving and update `output_bit_depth` variable.
|
||||
6. Proceed with merging, data type conversion, and saving.
|
||||
|
||||
**Phase 4: Configuration and Documentation**
|
||||
|
||||
1. **Modify `config.py`:** Ensure `RESOLUTION_THRESHOLD_FOR_JPG` is uncommented and set correctly (revert previous change).
|
||||
2. **Update `readme.md`:** Clarify the precedence: 8-bit maps >= threshold become JPG, otherwise the input-based logic applies.
|
||||
|
||||
## Visual Plan (Mermaid - Revised)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start] --> B(Phase 1: Enhance Classification - Done);
|
||||
B --> C(Phase 2: Modify _process_maps);
|
||||
C --> C1(Get original extension);
|
||||
C --> C2(Determine target bit depth);
|
||||
C --> C3(Get target_dim & threshold);
|
||||
C --> C4{8bit AND >= threshold?};
|
||||
C4 -- Yes --> C4a[Format=JPG];
|
||||
C4 -- No --> C5(Apply Input/Rule Logic);
|
||||
C5 -- force_8bit --> C5a[Format=PNG];
|
||||
C5 -- input=.jpg, 8bit --> C5b[Format=JPG];
|
||||
C5 -- input=.tif, 16bit --> C5c[Format=EXR/PNG (Config)];
|
||||
C5 -- input=.tif, 8bit --> C5d[Format=PNG];
|
||||
C5 -- input=other, 16bit --> C5c;
|
||||
C5 -- input=other, 8bit --> C5e[Format=PNG (Config)];
|
||||
C4a --> C6(Set Save Params & Save);
|
||||
C5a --> C6;
|
||||
C5b --> C6;
|
||||
C5c --> C6;
|
||||
C5d --> C6;
|
||||
C5e --> C6;
|
||||
C6 --> D(Phase 3: Modify _merge_maps);
|
||||
D --> D1(Get original extensions of inputs);
|
||||
D --> D2(Determine target bit depth from rule);
|
||||
D --> D3(Get target_dim & threshold);
|
||||
D --> D4{8bit AND >= threshold?};
|
||||
D4 -- Yes --> D4a[FinalFormat=JPG];
|
||||
D4 -- No --> D5(Apply Hierarchy Logic);
|
||||
D5 --> D5a(Find highest input format);
|
||||
D5a --> D5b{Highest = TIF?};
|
||||
D5b -- Yes --> D5c{Target 16bit?};
|
||||
D5c -- Yes --> D5d[FinalFormat=EXR/PNG (Config)];
|
||||
D5c -- No --> D5e[FinalFormat=PNG];
|
||||
D5b -- No --> D5f[FinalFormat=HighestInput];
|
||||
D4a --> D6(Set Save Params);
|
||||
D5d --> D6;
|
||||
D5e --> D6;
|
||||
D5f --> D6;
|
||||
D6 --> D7{Format=JPG AND Target=16bit?};
|
||||
D7 -- Yes --> D7a(Force data to 8bit);
|
||||
D7 -- No --> D8(Save Merged);
|
||||
D7a --> D8;
|
||||
D8 --> E(Phase 4: Config & Docs);
|
||||
E --> E1(Uncomment threshold in config.py);
|
||||
E --> E2(Update readme.md);
|
||||
E2 --> F[End];
|
||||
62
ProjectNotes/issue_definitions_editor_list_selection.md
Normal file
62
ProjectNotes/issue_definitions_editor_list_selection.md
Normal file
@ -0,0 +1,62 @@
|
||||
# Issue: List item selection not working in Definitions Editor
|
||||
|
||||
**Date:** 2025-05-13
|
||||
|
||||
**Affected File:** [`gui/definitions_editor_dialog.py`](gui/definitions_editor_dialog.py)
|
||||
|
||||
**Problem Description:**
|
||||
User mouse clicks on items within the `QListWidget` instances (for Asset Types, File Types, and Suppliers) in the Definitions Editor dialog do not trigger item selection or the `currentItemChanged` signal. The first item is selected by default and its details are displayed correctly. Programmatic selection of items (e.g., via a diagnostic button) *does* correctly trigger the `currentItemChanged` signal and updates the UI detail views. The issue is specific to user-initiated mouse clicks for selection after the initial load.
|
||||
|
||||
**Debugging Steps Taken & Findings:**
|
||||
|
||||
1. **Initial Analysis:**
|
||||
* Reviewed GUI internals documentation ([`Documentation/02_Developer_Guide/06_GUI_Internals.md`](Documentation/02_Developer_Guide/06_GUI_Internals.md)) and [`gui/definitions_editor_dialog.py`](gui/definitions_editor_dialog.py) source code.
|
||||
* Confirmed signal connections (`currentItemChanged` to display slots) are made.
|
||||
|
||||
2. **Logging in Display Slots (`_display_*_details`):**
|
||||
* Added logging to display slots. Confirmed they are called for the initial (default) item selection.
|
||||
* No further calls to these slots occur on user clicks, indicating `currentItemChanged` is not firing.
|
||||
|
||||
3. **Color Swatch Palette Role:**
|
||||
* Investigated and corrected `QPalette.ColorRole` for color swatches (reverted from `Background` to `Window`). This fixed an `AttributeError` but did not resolve the selection issue.
|
||||
|
||||
4. **Robust Error Handling in Display Slots:**
|
||||
* Wrapped display slot logic in `try...finally` blocks with detailed logging. Confirmed slots complete without error for initial selection and signals for detail widgets are reconnected.
|
||||
|
||||
5. **Diagnostic Lambda for `currentItemChanged`:**
|
||||
* Added a lambda logger to `currentItemChanged` alongside the main display slot.
|
||||
* Confirmed both lambda and display slot fire for initial programmatic selection.
|
||||
* Neither fires for subsequent user clicks. This proved the `QListWidget` itself was not emitting the signal.
|
||||
|
||||
6. **Explicit `setEnabled` and `setSelectionMode` on `QListWidget`:**
|
||||
* Explicitly set these properties. No change in behavior.
|
||||
|
||||
7. **Explicit `setEnabled` and `setFocusPolicy(Qt.ClickFocus)` on `tab_page` (parent of `QListWidget` layout):**
|
||||
* This change **allowed programmatic selection via a diagnostic button to correctly fire `currentItemChanged` and update the UI**.
|
||||
* However, user mouse clicks still did not work and did not fire the signal.
|
||||
|
||||
8. **Event Filter Investigation:**
|
||||
* **Filter on `QListWidget`:** Did NOT receive mouse press/release events from user clicks.
|
||||
* **Filter on `tab_page` (parent of `QListWidget`'s layout):** Did NOT receive mouse press/release events.
|
||||
* **Filter on `self.tab_widget` (QTabWidget):** DID receive mouse press/release events.
|
||||
* Modified `self.tab_widget`'s event filter to return `False` for events over the current page, attempting to ensure propagation.
|
||||
* **Result:** With the modified `tab_widget` filter, an event filter re-added to `asset_type_list_widget` *did* start receiving mouse press/release events. **However, `asset_type_list_widget` still did not emit `currentItemChanged` from these user clicks.**
|
||||
|
||||
9. **`DebugListWidget` (Subclassing `QListWidget`):**
|
||||
* Created `DebugListWidget` overriding `mousePressEvent` with logging.
|
||||
* Used `DebugListWidget` for `asset_type_list_widget`.
|
||||
* **Initial user report indicated that `DebugListWidget.mousePressEvent` logs were NOT appearing for user clicks.** This means that even with the `QTabWidget` event filter attempting to propagate events, and the `asset_type_list_widget`'s filter (from step 8) confirming it received them, the `mousePressEvent` of the `QListWidget` itself was not being triggered by those propagated events. This is the current mystery.
|
||||
|
||||
**Current Status:**
|
||||
- Programmatic selection works and fires signals.
|
||||
- User clicks are received by an event filter on `asset_type_list_widget` (after `QTabWidget` filter modification) but do not result in `mousePressEvent` being called on the `QListWidget` (or `DebugListWidget`) itself, and thus no `currentItemChanged` signal is emitted.
|
||||
- The issue seems to be a very low-level event processing problem specifically for user mouse clicks within the `QListWidget` instances when they are children of the `QTabWidget` pages, even when events appear to reach the list widget via an event filter.
|
||||
|
||||
**Next Steps (When Resuming):**
|
||||
1. Re-verify the logs from the `DebugListWidget.mousePressEvent` test. If it's truly not being called despite its event filter seeing events, this is extremely unusual.
|
||||
2. Simplify the `_create_tab_pane` method drastically for one tab:
|
||||
* Remove the right-hand pane.
|
||||
* Add the `DebugListWidget` directly to the `tab_page`'s layout without the intermediate `left_pane_layout`.
|
||||
3. Consider if any styles applied to `QListWidget` or its parents via stylesheets could be interfering with hit testing or event processing (unlikely for this specific symptom, but possible).
|
||||
4. Explore alternative ways to populate/manage the `QListWidget` or its items if a subtle corruption is occurring.
|
||||
5. If all else fails, consider replacing the `QListWidget` with a `QListView` and a `QStringListModel` as a more fundamental change to see if the issue is specific to `QListWidget` in this context.
|
||||
@ -1,73 +0,0 @@
|
||||
---
|
||||
title: Python Args and Kwargs - Python Cheatsheet
|
||||
description: args and kwargs may seem scary, but the truth is that they are not that difficult to grasp and have the power to grant your functions with flexibility and readability
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Args and Kwargs
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a href="https://docs.python.org/3/tutorial/index.html">Python args and kwargs Made Easy</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
<code>*args</code> and <code>**kwargs</code> may seem scary, but the truth is that they are not that difficult to grasp and have the power to grant your functions with lots of flexibility.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Read the article <router-link to="/blog/python-easy-args-kwargs">Python \*args and \*\*kwargs Made Easy</router-link> for a more in deep introduction.
|
||||
|
||||
## Args and Kwargs
|
||||
|
||||
`*args` and `**kwargs` allow you to pass an undefined number of arguments and keywords when calling a function.
|
||||
|
||||
```python
|
||||
>>> def some_function(*args, **kwargs):
|
||||
... pass
|
||||
...
|
||||
>>> # call some_function with any number of arguments
|
||||
>>> some_function(arg1, arg2, arg3)
|
||||
|
||||
>>> # call some_function with any number of keywords
|
||||
>>> some_function(key1=arg1, key2=arg2, key3=arg3)
|
||||
|
||||
>>> # call both, arguments and keywords
|
||||
>>> some_function(arg, key1=arg1)
|
||||
|
||||
>>> # or none
|
||||
>>> some_function()
|
||||
```
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
Python conventions
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
The words <code>*args</code> and <code>**kwargs</code> are conventions. They are not imposed by the interpreter, but considered good practice by the Python community.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
## args
|
||||
|
||||
You can access the _arguments_ through the `args` variable:
|
||||
|
||||
```python
|
||||
>>> def some_function(*args):
|
||||
... print(f'Arguments passed: {args} as {type(args)}')
|
||||
...
|
||||
>>> some_function('arg1', 'arg2', 'arg3')
|
||||
# Arguments passed: ('arg1', 'arg2', 'arg3') as <class 'tuple'>
|
||||
```
|
||||
|
||||
## kwargs
|
||||
|
||||
Keywords are accessed through the `kwargs` variable:
|
||||
|
||||
```python
|
||||
>>> def some_function(**kwargs):
|
||||
... print(f'keywords: {kwargs} as {type(kwargs)}')
|
||||
...
|
||||
>>> some_function(key1='arg1', key2='arg2')
|
||||
# keywords: {'key1': 'arg1', 'key2': 'arg2'} as <class 'dict'>
|
||||
```
|
||||
@ -1,340 +0,0 @@
|
||||
---
|
||||
title: Python Basics - Python Cheatsheet
|
||||
description: The basics of python. We all need to start somewhere, so how about doing it here.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Basics
|
||||
</base-title>
|
||||
|
||||
We all need to start somewhere, so how about doing it here.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the <a href="https://docs.python.org/3/tutorial/index.html">Python 3 tutorial</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Python is an easy to learn, powerful programming language [...] Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Math Operators
|
||||
|
||||
From **highest** to **lowest** precedence:
|
||||
|
||||
| Operators | Operation | Example |
|
||||
| --------- | ----------------- | --------------- |
|
||||
| \*\* | Exponent | `2 ** 3 = 8` |
|
||||
| % | Modulus/Remainder | `22 % 8 = 6` |
|
||||
| // | Integer division | `22 // 8 = 2` |
|
||||
| / | Division | `22 / 8 = 2.75` |
|
||||
| \* | Multiplication | `3 * 3 = 9` |
|
||||
| - | Subtraction | `5 - 2 = 3` |
|
||||
| + | Addition | `2 + 2 = 4` |
|
||||
|
||||
Examples of expressions:
|
||||
|
||||
```python
|
||||
>>> 2 + 3 * 6
|
||||
# 20
|
||||
|
||||
>>> (2 + 3) * 6
|
||||
# 30
|
||||
|
||||
>>> 2 ** 8
|
||||
#256
|
||||
|
||||
>>> 23 // 7
|
||||
# 3
|
||||
|
||||
>>> 23 % 7
|
||||
# 2
|
||||
|
||||
>>> (5 - 1) * ((7 + 1) / (3 - 1))
|
||||
# 16.0
|
||||
```
|
||||
|
||||
## Augmented Assignment Operators
|
||||
|
||||
| Operator | Equivalent |
|
||||
| ----------- | ---------------- |
|
||||
| `var += 1` | `var = var + 1` |
|
||||
| `var -= 1` | `var = var - 1` |
|
||||
| `var *= 1` | `var = var * 1` |
|
||||
| `var /= 1` | `var = var / 1` |
|
||||
| `var //= 1` | `var = var // 1` |
|
||||
| `var %= 1` | `var = var % 1` |
|
||||
| `var **= 1` | `var = var ** 1` |
|
||||
|
||||
Examples:
|
||||
|
||||
```python
|
||||
>>> greeting = 'Hello'
|
||||
>>> greeting += ' world!'
|
||||
>>> greeting
|
||||
# 'Hello world!'
|
||||
|
||||
>>> number = 1
|
||||
>>> number += 1
|
||||
>>> number
|
||||
# 2
|
||||
|
||||
>>> my_list = ['item']
|
||||
>>> my_list *= 3
|
||||
>>> my_list
|
||||
# ['item', 'item', 'item']
|
||||
```
|
||||
|
||||
## Walrus Operator
|
||||
|
||||
The Walrus Operator allows assignment of variables within an expression while returning the value of the variable
|
||||
|
||||
Example:
|
||||
|
||||
```python
|
||||
>>> print(my_var:="Hello World!")
|
||||
# 'Hello world!'
|
||||
|
||||
>>> my_var="Yes"
|
||||
>>> print(my_var)
|
||||
# 'Yes'
|
||||
|
||||
>>> print(my_var:="Hello")
|
||||
# 'Hello'
|
||||
```
|
||||
|
||||
The _Walrus Operator_, or **Assignment Expression Operator** was firstly introduced in 2018 via [PEP 572](https://peps.python.org/pep-0572/), and then officially released with **Python 3.8** in October 2019.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Syntax Semantics & Examples
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The <a href="https://peps.python.org/pep-0572/" target="_blank">PEP 572</a> provides the syntax, semantics and examples for the Walrus Operator.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Data Types
|
||||
|
||||
| Data Type | Examples |
|
||||
| ---------------------- | ----------------------------------------- |
|
||||
| Integers | `-2, -1, 0, 1, 2, 3, 4, 5` |
|
||||
| Floating-point numbers | `-1.25, -1.0, --0.5, 0.0, 0.5, 1.0, 1.25` |
|
||||
| Strings | `'a', 'aa', 'aaa', 'Hello!', '11 cats'` |
|
||||
|
||||
## Concatenation and Replication
|
||||
|
||||
String concatenation:
|
||||
|
||||
```python
|
||||
>>> 'Alice' 'Bob'
|
||||
# 'AliceBob'
|
||||
```
|
||||
|
||||
String replication:
|
||||
|
||||
```python
|
||||
>>> 'Alice' * 5
|
||||
# 'AliceAliceAliceAliceAlice'
|
||||
```
|
||||
|
||||
## Variables
|
||||
|
||||
You can name a variable anything as long as it obeys the following rules:
|
||||
|
||||
1. It can be only one word.
|
||||
|
||||
```python
|
||||
>>> # bad
|
||||
>>> my variable = 'Hello'
|
||||
|
||||
>>> # good
|
||||
>>> var = 'Hello'
|
||||
```
|
||||
|
||||
2. It can use only letters, numbers, and the underscore (`_`) character.
|
||||
|
||||
```python
|
||||
>>> # bad
|
||||
>>> %$@variable = 'Hello'
|
||||
|
||||
>>> # good
|
||||
>>> my_var = 'Hello'
|
||||
|
||||
>>> # good
|
||||
>>> my_var_2 = 'Hello'
|
||||
```
|
||||
|
||||
3. It can’t begin with a number.
|
||||
|
||||
```python
|
||||
>>> # this wont work
|
||||
>>> 23_var = 'hello'
|
||||
```
|
||||
|
||||
4. Variable name starting with an underscore (`_`) are considered as "unuseful".
|
||||
|
||||
```python
|
||||
>>> # _spam should not be used again in the code
|
||||
>>> _spam = 'Hello'
|
||||
```
|
||||
|
||||
## Comments
|
||||
|
||||
Inline comment:
|
||||
|
||||
```python
|
||||
# This is a comment
|
||||
```
|
||||
|
||||
Multiline comment:
|
||||
|
||||
```python
|
||||
# This is a
|
||||
# multiline comment
|
||||
```
|
||||
|
||||
Code with a comment:
|
||||
|
||||
```python
|
||||
a = 1 # initialization
|
||||
```
|
||||
|
||||
Please note the two spaces in front of the comment.
|
||||
|
||||
Function docstring:
|
||||
|
||||
```python
|
||||
def foo():
|
||||
"""
|
||||
This is a function docstring
|
||||
You can also use:
|
||||
''' Function Docstring '''
|
||||
"""
|
||||
```
|
||||
|
||||
## The print() Function
|
||||
|
||||
The `print()` function writes the value of the argument(s) it is given. [...] it handles multiple arguments, floating point-quantities, and strings. Strings are printed without quotes, and a space is inserted between items, so you can format things nicely:
|
||||
|
||||
```python
|
||||
>>> print('Hello world!')
|
||||
# Hello world!
|
||||
|
||||
>>> a = 1
|
||||
>>> print('Hello world!', a)
|
||||
# Hello world! 1
|
||||
```
|
||||
|
||||
### The end keyword
|
||||
|
||||
The keyword argument `end` can be used to avoid the newline after the output, or end the output with a different string:
|
||||
|
||||
```python
|
||||
phrase = ['printed', 'with', 'a', 'dash', 'in', 'between']
|
||||
>>> for word in phrase:
|
||||
... print(word, end='-')
|
||||
...
|
||||
# printed-with-a-dash-in-between-
|
||||
```
|
||||
|
||||
### The sep keyword
|
||||
|
||||
The keyword `sep` specify how to separate the objects, if there is more than one:
|
||||
|
||||
```python
|
||||
print('cats', 'dogs', 'mice', sep=',')
|
||||
# cats,dogs,mice
|
||||
```
|
||||
|
||||
## The input() Function
|
||||
|
||||
This function takes the input from the user and converts it into a string:
|
||||
|
||||
```python
|
||||
>>> print('What is your name?') # ask for their name
|
||||
>>> my_name = input()
|
||||
>>> print('Hi, {}'.format(my_name))
|
||||
# What is your name?
|
||||
# Martha
|
||||
# Hi, Martha
|
||||
```
|
||||
|
||||
`input()` can also set a default message without using `print()`:
|
||||
|
||||
```python
|
||||
>>> my_name = input('What is your name? ') # default message
|
||||
>>> print('Hi, {}'.format(my_name))
|
||||
# What is your name? Martha
|
||||
# Hi, Martha
|
||||
```
|
||||
|
||||
It is also possible to use formatted strings to avoid using .format:
|
||||
|
||||
```python
|
||||
>>> my_name = input('What is your name? ') # default message
|
||||
>>> print(f'Hi, {my_name}')
|
||||
# What is your name? Martha
|
||||
# Hi, Martha
|
||||
```
|
||||
|
||||
|
||||
## The len() Function
|
||||
|
||||
Evaluates to the integer value of the number of characters in a string, list, dictionary, etc.:
|
||||
|
||||
```python
|
||||
>>> len('hello')
|
||||
# 5
|
||||
|
||||
>>> len(['cat', 3, 'dog'])
|
||||
# 3
|
||||
```
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>Test of emptiness</base-warning-title>
|
||||
<base-warning-content>
|
||||
Test of emptiness of strings, lists, dictionaries, etc., should not use
|
||||
<code>len</code>, but prefer direct boolean evaluation.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
Test of emptiness example:
|
||||
|
||||
```python
|
||||
>>> a = [1, 2, 3]
|
||||
|
||||
# bad
|
||||
>>> if len(a) > 0: # evaluates to True
|
||||
... print("the list is not empty!")
|
||||
...
|
||||
# the list is not empty!
|
||||
|
||||
# good
|
||||
>>> if a: # evaluates to True
|
||||
... print("the list is not empty!")
|
||||
...
|
||||
# the list is not empty!
|
||||
```
|
||||
|
||||
## The str(), int(), and float() Functions
|
||||
|
||||
These functions allow you to change the type of variable. For example, you can transform from an `integer` or `float` to a `string`:
|
||||
|
||||
```python
|
||||
>>> str(29)
|
||||
# '29'
|
||||
|
||||
>>> str(-3.14)
|
||||
# '-3.14'
|
||||
```
|
||||
|
||||
Or from a `string` to an `integer` or `float`:
|
||||
|
||||
```python
|
||||
>>> int('11')
|
||||
# 11
|
||||
|
||||
>>> float('3.14')
|
||||
# 3.14
|
||||
```
|
||||
@ -1,83 +0,0 @@
|
||||
---
|
||||
title: Python built-in functions - Python Cheatsheet
|
||||
description: The Python interpreter has a number of functions and types built into it that are always available.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Built-in Functions
|
||||
</base-title>
|
||||
|
||||
The Python interpreter has a number of functions and types built into it that are always available.
|
||||
|
||||
## Python built-in Functions
|
||||
|
||||
| Function | Description |
|
||||
| -------------------------------------------------------------------- | ------------------------------------------------------------------------- |
|
||||
| <router-link to='/builtin/abs'>abs()</router-link> | Return the absolute value of a number. |
|
||||
| <router-link to='/builtin/aiter'>aiter()</router-link> | Return an asynchronous iterator for an asynchronous iterable. |
|
||||
| <router-link to='/builtin/all'>all()</router-link> | Return True if all elements of the iterable are true. |
|
||||
| <router-link to='/builtin/any'>any()</router-link> | Return True if any element of the iterable is true. |
|
||||
| <router-link to='/builtin/ascii'>ascii()</router-link> | Return a string with a printable representation of an object. |
|
||||
| <router-link to='/builtin/bin'>bin()</router-link> | Convert an integer number to a binary string. |
|
||||
| <router-link to='/builtin/bool'>bool()</router-link> | Return a Boolean value. |
|
||||
| <router-link to='/builtin/breakpoint'>breakpoint()</router-link> | Drops you into the debugger at the call site. |
|
||||
| <router-link to='/builtin/bytearray'>bytearray()</router-link> | Return a new array of bytes. |
|
||||
| <router-link to='/builtin/bytes'>bytes()</router-link> | Return a new “bytes” object. |
|
||||
| <router-link to='/builtin/callable'>callable()</router-link> | Return True if the object argument is callable, False if not. |
|
||||
| <router-link to='/builtin/chr'>chr()</router-link> | Return the string representing a character. |
|
||||
| <router-link to='/builtin/classmethod'>classmethod()</router-link> | Transform a method into a class method. |
|
||||
| <router-link to='/builtin/compile'>compile()</router-link> | Compile the source into a code or AST object. |
|
||||
| <router-link to='/builtin/complex'>complex()</router-link> | Return a complex number with the value real + imag\*1j. |
|
||||
| <router-link to='/builtin/delattr'>delattr()</router-link> | Deletes the named attribute, provided the object allows it. |
|
||||
| <router-link to='/builtin/dict'>dict()</router-link> | Create a new dictionary. |
|
||||
| <router-link to='/builtin/dir'>dir()</router-link> | Return the list of names in the current local scope. |
|
||||
| <router-link to='/builtin/divmod'>divmod()</router-link> | Return a pair of numbers consisting of their quotient and remainder. |
|
||||
| <router-link to='/builtin/enumerate'>enumerate()</router-link> | Return an enumerate object. |
|
||||
| <router-link to='/builtin/eval'>eval()</router-link> | Evaluates and executes an expression. |
|
||||
| <router-link to='/builtin/exec'>exec()</router-link> | This function supports dynamic execution of Python code. |
|
||||
| <router-link to='/builtin/filter'>filter()</router-link> | Construct an iterator from an iterable and returns true. |
|
||||
| <router-link to='/builtin/float'>float()</router-link> | Return a floating point number from a number or string. |
|
||||
| <router-link to='/builtin/format'>format()</router-link> | Convert a value to a “formatted” representation. |
|
||||
| <router-link to='/builtin/frozenset'>frozenset()</router-link> | Return a new frozenset object. |
|
||||
| <router-link to='/builtin/getattr'>getattr()</router-link> | Return the value of the named attribute of object. |
|
||||
| <router-link to='/builtin/globals'>globals()</router-link> | Return the dictionary implementing the current module namespace. |
|
||||
| <router-link to='/builtin/hasattr'>hasattr()</router-link> | True if the string is the name of one of the object’s attributes. |
|
||||
| <router-link to='/builtin/hash'>hash()</router-link> | Return the hash value of the object. |
|
||||
| <router-link to='/builtin/help'>help()</router-link> | Invoke the built-in help system. |
|
||||
| <router-link to='/builtin/hex'>hex()</router-link> | Convert an integer number to a lowercase hexadecimal string. |
|
||||
| <router-link to='/builtin/id'>id()</router-link> | Return the “identity” of an object. |
|
||||
| <router-link to='/builtin/input'>input()</router-link> | This function takes an input and converts it into a string. |
|
||||
| <router-link to='/builtin/int'>int()</router-link> | Return an integer object constructed from a number or string. |
|
||||
| <router-link to='/builtin/isinstance'>isinstance()</router-link> | Return True if the object argument is an instance of an object. |
|
||||
| <router-link to='/builtin/issubclass'>issubclass()</router-link> | Return True if class is a subclass of classinfo. |
|
||||
| <router-link to='/builtin/iter'>iter()</router-link> | Return an iterator object. |
|
||||
| <router-link to='/builtin/len'>len()</router-link> | Return the length (the number of items) of an object. |
|
||||
| <router-link to='/builtin/list'>list()</router-link> | Rather than being a function, list is a mutable sequence type. |
|
||||
| <router-link to='/builtin/locals'>locals()</router-link> | Update and return a dictionary with the current local symbol table. |
|
||||
| <router-link to='/builtin/map'>map()</router-link> | Return an iterator that applies function to every item of iterable. |
|
||||
| <router-link to='/builtin/max'>max()</router-link> | Return the largest item in an iterable. |
|
||||
| <router-link to='/builtin/min'>min()</router-link> | Return the smallest item in an iterable. |
|
||||
| <router-link to='/builtin/next'>next()</router-link> | Retrieve the next item from the iterator. |
|
||||
| <router-link to='/builtin/object'>object()</router-link> | Return a new featureless object. |
|
||||
| <router-link to='/builtin/oct'>oct()</router-link> | Convert an integer number to an octal string. |
|
||||
| <router-link to='/builtin/open'>open()</router-link> | Open file and return a corresponding file object. |
|
||||
| <router-link to='/builtin/ord'>ord()</router-link> | Return an integer representing the Unicode code point of a character. |
|
||||
| <router-link to='/builtin/pow'>pow()</router-link> | Return base to the power exp. |
|
||||
| <router-link to='/builtin/print'>print()</router-link> | Print objects to the text stream file. |
|
||||
| <router-link to='/builtin/property'>property()</router-link> | Return a property attribute. |
|
||||
| <router-link to='/builtin/repr'>repr()</router-link> | Return a string containing a printable representation of an object. |
|
||||
| <router-link to='/builtin/reversed'>reversed()</router-link> | Return a reverse iterator. |
|
||||
| <router-link to='/builtin/round'>round()</router-link> | Return number rounded to ndigits precision after the decimal point. |
|
||||
| <router-link to='/builtin/set'>set()</router-link> | Return a new set object. |
|
||||
| <router-link to='/builtin/setattr'>setattr()</router-link> | This is the counterpart of getattr(). |
|
||||
| <router-link to='/builtin/slice'>slice()</router-link> | Return a sliced object representing a set of indices. |
|
||||
| <router-link to='/builtin/sorted'>sorted()</router-link> | Return a new sorted list from the items in iterable. |
|
||||
| <router-link to='/builtin/staticmethod'>staticmethod()</router-link> | Transform a method into a static method. |
|
||||
| <router-link to='/builtin/str'>str()</router-link> | Return a str version of object. |
|
||||
| <router-link to='/builtin/sum'>sum()</router-link> | Sums start and the items of an iterable. |
|
||||
| <router-link to='/builtin/super'>super()</router-link> | Return a proxy object that delegates method calls to a parent or sibling. |
|
||||
| <router-link to='/builtin/tuple'>tuple()</router-link> | Rather than being a function, is actually an immutable sequence type. |
|
||||
| <router-link to='/builtin/type'>type()</router-link> | Return the type of an object. |
|
||||
| <router-link to='/builtin/vars'>vars()</router-link> | Return the dict attribute for any other object with a dict attribute. |
|
||||
| <router-link to='/builtin/zip'>zip()</router-link> | Iterate over several iterables in parallel. |
|
||||
| <router-link to='/builtin/import'>**import**()</router-link> | This function is invoked by the import statement. |
|
||||
@ -1,120 +0,0 @@
|
||||
---
|
||||
title: Python Comprehensions - Python Cheatsheet
|
||||
description: List comprehensions provide a concise way to create lists
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Comprehensions
|
||||
</base-title>
|
||||
|
||||
List Comprehensions are a special kind of syntax that let us create lists out of other lists, and are incredibly useful when dealing with numbers and with one or two levels of nested for loops.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the Python 3 <a target="_blank" href="https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions">tutorial</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
List comprehensions provide a concise way to create lists. [...] or to create a subsequence of those elements that satisfy a certain condition.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Read <router-link to="/blog/python-comprehensions-step-by-step">Python Comprehensions: A step by step Introduction</router-link> for a more in-depth introduction.
|
||||
|
||||
## List comprehension
|
||||
|
||||
This is how we create a new list from an existing collection with a For Loop:
|
||||
|
||||
```python
|
||||
>>> names = ['Charles', 'Susan', 'Patrick', 'George']
|
||||
|
||||
>>> new_list = []
|
||||
>>> for n in names:
|
||||
... new_list.append(n)
|
||||
...
|
||||
>>> new_list
|
||||
# ['Charles', 'Susan', 'Patrick', 'George']
|
||||
```
|
||||
|
||||
And this is how we do the same with a List Comprehension:
|
||||
|
||||
```python
|
||||
>>> names = ['Charles', 'Susan', 'Patrick', 'George']
|
||||
|
||||
>>> new_list = [n for n in names]
|
||||
>>> new_list
|
||||
# ['Charles', 'Susan', 'Patrick', 'George']
|
||||
```
|
||||
|
||||
We can do the same with numbers:
|
||||
|
||||
```python
|
||||
>>> n = [(a, b) for a in range(1, 3) for b in range(1, 3)]
|
||||
>>> n
|
||||
# [(1, 1), (1, 2), (2, 1), (2, 2)]
|
||||
```
|
||||
|
||||
## Adding conditionals
|
||||
|
||||
If we want `new_list` to have only the names that start with C, with a for loop, we would do it like this:
|
||||
|
||||
```python
|
||||
>>> names = ['Charles', 'Susan', 'Patrick', 'George', 'Carol']
|
||||
|
||||
>>> new_list = []
|
||||
>>> for n in names:
|
||||
... if n.startswith('C'):
|
||||
... new_list.append(n)
|
||||
...
|
||||
>>> print(new_list)
|
||||
# ['Charles', 'Carol']
|
||||
```
|
||||
|
||||
In a List Comprehension, we add the `if` statement at the end:
|
||||
|
||||
```python
|
||||
>>> new_list = [n for n in names if n.startswith('C')]
|
||||
>>> print(new_list)
|
||||
# ['Charles', 'Carol']
|
||||
```
|
||||
|
||||
To use an `if-else` statement in a List Comprehension:
|
||||
|
||||
```python
|
||||
>>> nums = [1, 2, 3, 4, 5, 6]
|
||||
>>> new_list = [num*2 if num % 2 == 0 else num for num in nums]
|
||||
>>> print(new_list)
|
||||
# [1, 4, 3, 8, 5, 12]
|
||||
```
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Set and Dict comprehensions
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The basics of `list` comprehensions also apply to <b>sets</b> and <b>dictionaries</b>.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Set comprehension
|
||||
|
||||
```python
|
||||
>>> b = {"abc", "def"}
|
||||
>>> {s.upper() for s in b}
|
||||
{"ABC", "DEF"}
|
||||
```
|
||||
|
||||
## Dict comprehension
|
||||
|
||||
```python
|
||||
>>> c = {'name': 'Pooka', 'age': 5}
|
||||
>>> {v: k for k, v in c.items()}
|
||||
{'Pooka': 'name', 5: 'age'}
|
||||
```
|
||||
|
||||
A List comprehension can be generated from a dictionary:
|
||||
|
||||
```python
|
||||
>>> c = {'name': 'Pooka', 'age': 5}
|
||||
>>> ["{}:{}".format(k.upper(), v) for k, v in c.items()]
|
||||
['NAME:Pooka', 'AGE:5']
|
||||
```
|
||||
@ -1,68 +0,0 @@
|
||||
---
|
||||
title: Python Context Manager - Python Cheatsheet
|
||||
description: While Python's context managers are widely used, few understand the purpose behind their use. These statements, commonly used with reading and writing files, assist the application in conserving system memory and improve resource management by ensuring specific resources are only in use for certain processes.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Context Manager
|
||||
</base-title>
|
||||
|
||||
While Python's context managers are widely used, few understand the purpose behind their use. These statements, commonly used with reading and writing files, assist the application in conserving system memory and improve resource management by ensuring specific resources are only in use for certain processes.
|
||||
|
||||
## The with statement
|
||||
|
||||
A context manager is an object that is notified when a context (a block of code) starts and ends. You commonly use one with the `with` statement. It takes care of the notifying.
|
||||
|
||||
For example, file objects are context managers. When a context ends, the file object is closed automatically:
|
||||
|
||||
```python
|
||||
>>> with open(filename) as f:
|
||||
... file_contents = f.read()
|
||||
...
|
||||
>>> # the open_file object has automatically been closed.
|
||||
```
|
||||
|
||||
Anything that ends execution of the block causes the context manager's exit method to be called. This includes exceptions, and can be useful when an error causes you to prematurely exit an open file or connection. Exiting a script without properly closing files/connections is a bad idea, that may cause data loss or other problems. By using a context manager, you can ensure that precautions are always taken to prevent damage or loss in this way.
|
||||
|
||||
## Writing your own context manager
|
||||
|
||||
It is also possible to write a context manager using generator syntax thanks to the `contextlib.contextmanager` decorator:
|
||||
|
||||
```python
|
||||
>>> import contextlib
|
||||
>>> @contextlib.contextmanager
|
||||
... def context_manager(num):
|
||||
... print('Enter')
|
||||
... yield num + 1
|
||||
... print('Exit')
|
||||
...
|
||||
>>> with context_manager(2) as cm:
|
||||
... # the following instructions are run when
|
||||
... # the 'yield' point of the context manager is
|
||||
... # reached. 'cm' will have the value that was yielded
|
||||
... print('Right in the middle with cm = {}'.format(cm))
|
||||
...
|
||||
# Enter
|
||||
# Right in the middle with cm = 3
|
||||
# Exit
|
||||
```
|
||||
|
||||
|
||||
## Class based context manager
|
||||
|
||||
You can define class based context manager. The key methods are `__enter__` and `__exit__`
|
||||
```python
|
||||
class ContextManager:
|
||||
def __enter__(self, *args, **kwargs):
|
||||
print("--enter--")
|
||||
|
||||
def __exit__(self, *args):
|
||||
print("--exit--")
|
||||
|
||||
|
||||
with ContextManager():
|
||||
print("test")
|
||||
#--enter--
|
||||
#test
|
||||
#--exit--
|
||||
```
|
||||
@ -1,490 +0,0 @@
|
||||
---
|
||||
title: Python Control Flow - Python Cheatsheet
|
||||
description: Control flow is the order in which individual statements, instructions or function calls are executed or evaluated. The control flow of a Python program is regulated by conditional statements, loops, and function calls.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Control Flow
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Python control flow
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Control flow is the order in which individual statements, instructions, or function calls are executed or evaluated. The control flow of a Python program is regulated by conditional statements, loops, and function calls.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Comparison Operators
|
||||
|
||||
| Operator | Meaning |
|
||||
| -------- | ------------------------ |
|
||||
| `==` | Equal to |
|
||||
| `!=` | Not equal to |
|
||||
| `<` | Less than |
|
||||
| `>` | Greater Than |
|
||||
| `<=` | Less than or Equal to |
|
||||
| `>=` | Greater than or Equal to |
|
||||
|
||||
These operators evaluate to True or False depending on the values you give them.
|
||||
|
||||
Examples:
|
||||
|
||||
```python
|
||||
>>> 42 == 42
|
||||
True
|
||||
|
||||
>>> 40 == 42
|
||||
False
|
||||
|
||||
>>> 'hello' == 'hello'
|
||||
True
|
||||
|
||||
>>> 'hello' == 'Hello'
|
||||
False
|
||||
|
||||
>>> 'dog' != 'cat'
|
||||
True
|
||||
|
||||
>>> 42 == 42.0
|
||||
True
|
||||
|
||||
>>> 42 == '42'
|
||||
False
|
||||
```
|
||||
|
||||
## Boolean Operators
|
||||
|
||||
There are three Boolean operators: `and`, `or`, and `not`.
|
||||
In the order of precedence, highest to lowest they are `not`, `and` and `or`.
|
||||
|
||||
The `and` Operator’s _Truth_ Table:
|
||||
|
||||
| Expression | Evaluates to |
|
||||
| ----------------- | ------------ |
|
||||
| `True and True` | `True` |
|
||||
| `True and False` | `False` |
|
||||
| `False and True` | `False` |
|
||||
| `False and False` | `False` |
|
||||
|
||||
The `or` Operator’s _Truth_ Table:
|
||||
|
||||
| Expression | Evaluates to |
|
||||
| ---------------- | ------------ |
|
||||
| `True or True` | `True` |
|
||||
| `True or False` | `True` |
|
||||
| `False or True` | `True` |
|
||||
| `False or False` | `False` |
|
||||
|
||||
The `not` Operator’s _Truth_ Table:
|
||||
|
||||
| Expression | Evaluates to |
|
||||
| ----------- | ------------ |
|
||||
| `not True` | `False` |
|
||||
| `not False` | `True` |
|
||||
|
||||
## Mixing Operators
|
||||
|
||||
You can mix boolean and comparison operators:
|
||||
|
||||
```python
|
||||
>>> (4 < 5) and (5 < 6)
|
||||
True
|
||||
|
||||
>>> (4 < 5) and (9 < 6)
|
||||
False
|
||||
|
||||
>>> (1 == 2) or (2 == 2)
|
||||
True
|
||||
```
|
||||
|
||||
Also, you can mix use multiple Boolean operators in an expression, along with the comparison operators:
|
||||
|
||||
```python
|
||||
>>> 2 + 2 == 4 and not 2 + 2 == 5 and 2 * 2 == 2 + 2
|
||||
True
|
||||
>>> # In the statement below 3 < 4 and 5 > 5 gets executed first evaluating to False
|
||||
>>> # Then 5 > 4 returns True so the results after True or False is True
|
||||
>>> 5 > 4 or 3 < 4 and 5 > 5
|
||||
True
|
||||
>>> # Now the statement within parentheses gets executed first so True and False returns False.
|
||||
>>> (5 > 4 or 3 < 4) and 5 > 5
|
||||
False
|
||||
```
|
||||
|
||||
## if Statements
|
||||
|
||||
The `if` statement evaluates an expression, and if that expression is `True`, it then executes the following indented code:
|
||||
|
||||
```python
|
||||
>>> name = 'Debora'
|
||||
|
||||
>>> if name == 'Debora':
|
||||
... print('Hi, Debora')
|
||||
...
|
||||
# Hi, Debora
|
||||
|
||||
>>> if name != 'George':
|
||||
... print('You are not George')
|
||||
...
|
||||
# You are not George
|
||||
```
|
||||
|
||||
The `else` statement executes only if the evaluation of the `if` and all the `elif` expressions are `False`:
|
||||
|
||||
```python
|
||||
>>> name = 'Debora'
|
||||
|
||||
>>> if name == 'George':
|
||||
... print('Hi, George.')
|
||||
... else:
|
||||
... print('You are not George')
|
||||
...
|
||||
# You are not George
|
||||
```
|
||||
|
||||
Only after the `if` statement expression is `False`, the `elif` statement is evaluated and executed:
|
||||
|
||||
```python
|
||||
>>> name = 'George'
|
||||
|
||||
>>> if name == 'Debora':
|
||||
... print('Hi Debora!')
|
||||
... elif name == 'George':
|
||||
... print('Hi George!')
|
||||
...
|
||||
# Hi George!
|
||||
```
|
||||
|
||||
the `elif` and `else` parts are optional.
|
||||
|
||||
```python
|
||||
>>> name = 'Antony'
|
||||
|
||||
>>> if name == 'Debora':
|
||||
... print('Hi Debora!')
|
||||
... elif name == 'George':
|
||||
... print('Hi George!')
|
||||
... else:
|
||||
... print('Who are you?')
|
||||
...
|
||||
# Who are you?
|
||||
```
|
||||
|
||||
## Ternary Conditional Operator
|
||||
|
||||
Many programming languages have a ternary operator, which define a conditional expression. The most common usage is to make a terse, simple conditional assignment statement. In other words, it offers one-line code to evaluate the first expression if the condition is true, and otherwise it evaluates the second expression.
|
||||
|
||||
```
|
||||
<expression1> if <condition> else <expression2>
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```python
|
||||
>>> age = 15
|
||||
|
||||
>>> # this if statement:
|
||||
>>> if age < 18:
|
||||
... print('kid')
|
||||
... else:
|
||||
... print('adult')
|
||||
...
|
||||
# output: kid
|
||||
|
||||
>>> # is equivalent to this ternary operator:
|
||||
>>> print('kid' if age < 18 else 'adult')
|
||||
# output: kid
|
||||
```
|
||||
|
||||
Ternary operators can be chained:
|
||||
|
||||
```python
|
||||
>>> age = 15
|
||||
|
||||
>>> # this ternary operator:
|
||||
>>> print('kid' if age < 13 else 'teen' if age < 18 else 'adult')
|
||||
|
||||
>>> # is equivalent to this if statement:
|
||||
>>> if age < 18:
|
||||
... if age < 13:
|
||||
... print('kid')
|
||||
... else:
|
||||
... print('teen')
|
||||
... else:
|
||||
... print('adult')
|
||||
...
|
||||
# output: teen
|
||||
```
|
||||
|
||||
## Switch-Case Statement
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Switch-Case statements
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
In computer programming languages, a switch statement is a type of selection control mechanism used to allow the value of a variable or expression to change the control flow of program execution via search and map.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
The _Switch-Case statements_, or **Structural Pattern Matching**, was firstly introduced in 2020 via [PEP 622](https://peps.python.org/pep-0622/), and then officially released with **Python 3.10** in September 2022.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Official Tutorial
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The <a href="https://peps.python.org/pep-0636/" target="_blank">PEP 636</a> provides an official tutorial for the Python Pattern matching or Switch-Case statements.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
### Matching single values
|
||||
|
||||
```python
|
||||
>>> response_code = 201
|
||||
>>> match response_code:
|
||||
... case 200:
|
||||
... print("OK")
|
||||
... case 201:
|
||||
... print("Created")
|
||||
... case 300:
|
||||
... print("Multiple Choices")
|
||||
... case 307:
|
||||
... print("Temporary Redirect")
|
||||
... case 404:
|
||||
... print("404 Not Found")
|
||||
... case 500:
|
||||
... print("Internal Server Error")
|
||||
... case 502:
|
||||
... print("502 Bad Gateway")
|
||||
...
|
||||
# Created
|
||||
```
|
||||
|
||||
### Matching with the or Pattern
|
||||
|
||||
In this example, the pipe character (`|` or `or`) allows python to return the same response for two or more cases.
|
||||
|
||||
```python
|
||||
>>> response_code = 502
|
||||
>>> match response_code:
|
||||
... case 200 | 201:
|
||||
... print("OK")
|
||||
... case 300 | 307:
|
||||
... print("Redirect")
|
||||
... case 400 | 401:
|
||||
... print("Bad Request")
|
||||
... case 500 | 502:
|
||||
... print("Internal Server Error")
|
||||
...
|
||||
# Internal Server Error
|
||||
```
|
||||
|
||||
### Matching by the length of an Iterable
|
||||
|
||||
```python
|
||||
>>> today_responses = [200, 300, 404, 500]
|
||||
>>> match today_responses:
|
||||
... case [a]:
|
||||
... print(f"One response today: {a}")
|
||||
... case [a, b]:
|
||||
... print(f"Two responses today: {a} and {b}")
|
||||
... case [a, b, *rest]:
|
||||
... print(f"All responses: {a}, {b}, {rest}")
|
||||
...
|
||||
# All responses: 200, 300, [404, 500]
|
||||
```
|
||||
|
||||
### Default value
|
||||
|
||||
The underscore symbol (`_`) is used to define a default case:
|
||||
|
||||
```python
|
||||
>>> response_code = 800
|
||||
>>> match response_code:
|
||||
... case 200 | 201:
|
||||
... print("OK")
|
||||
... case 300 | 307:
|
||||
... print("Redirect")
|
||||
... case 400 | 401:
|
||||
... print("Bad Request")
|
||||
... case 500 | 502:
|
||||
... print("Internal Server Error")
|
||||
... case _:
|
||||
... print("Invalid Code")
|
||||
...
|
||||
# Invalid Code
|
||||
```
|
||||
|
||||
### Matching Builtin Classes
|
||||
|
||||
```python
|
||||
>>> response_code = "300"
|
||||
>>> match response_code:
|
||||
... case int():
|
||||
... print('Code is a number')
|
||||
... case str():
|
||||
... print('Code is a string')
|
||||
... case _:
|
||||
... print('Code is neither a string nor a number')
|
||||
...
|
||||
# Code is a string
|
||||
```
|
||||
|
||||
### Guarding Match-Case Statements
|
||||
|
||||
```python
|
||||
>>> response_code = 300
|
||||
>>> match response_code:
|
||||
... case int():
|
||||
... if response_code > 99 and response_code < 500:
|
||||
... print('Code is a valid number')
|
||||
... case _:
|
||||
... print('Code is an invalid number')
|
||||
...
|
||||
# Code is a valid number
|
||||
```
|
||||
|
||||
## while Loop Statements
|
||||
|
||||
The while statement is used for repeated execution as long as an expression is `True`:
|
||||
|
||||
```python
|
||||
>>> spam = 0
|
||||
>>> while spam < 5:
|
||||
... print('Hello, world.')
|
||||
... spam = spam + 1
|
||||
...
|
||||
# Hello, world.
|
||||
# Hello, world.
|
||||
# Hello, world.
|
||||
# Hello, world.
|
||||
# Hello, world.
|
||||
```
|
||||
|
||||
## break Statements
|
||||
|
||||
If the execution reaches a `break` statement, it immediately exits the `while` loop’s clause:
|
||||
|
||||
```python
|
||||
>>> while True:
|
||||
... name = input('Please type your name: ')
|
||||
... if name == 'your name':
|
||||
... break
|
||||
...
|
||||
>>> print('Thank you!')
|
||||
# Please type your name: your name
|
||||
# Thank you!
|
||||
```
|
||||
|
||||
## continue Statements
|
||||
|
||||
When the program execution reaches a `continue` statement, the program execution immediately jumps back to the start of the loop.
|
||||
|
||||
```python
|
||||
>>> while True:
|
||||
... name = input('Who are you? ')
|
||||
... if name != 'Joe':
|
||||
... continue
|
||||
... password = input('Password? (It is a fish.): ')
|
||||
... if password == 'swordfish':
|
||||
... break
|
||||
...
|
||||
>>> print('Access granted.')
|
||||
# Who are you? Charles
|
||||
# Who are you? Debora
|
||||
# Who are you? Joe
|
||||
# Password? (It is a fish.): swordfish
|
||||
# Access granted.
|
||||
```
|
||||
|
||||
## For loop
|
||||
|
||||
The `for` loop iterates over a `list`, `tuple`, `dictionary`, `set` or `string`:
|
||||
|
||||
```python
|
||||
>>> pets = ['Bella', 'Milo', 'Loki']
|
||||
>>> for pet in pets:
|
||||
... print(pet)
|
||||
...
|
||||
# Bella
|
||||
# Milo
|
||||
# Loki
|
||||
```
|
||||
|
||||
## The range() function
|
||||
|
||||
The `range()` function returns a sequence of numbers. It starts from 0, increments by 1, and stops before a specified number:
|
||||
|
||||
```python
|
||||
>>> for i in range(5):
|
||||
... print(f'Will stop at 5! or 4? ({i})')
|
||||
...
|
||||
# Will stop at 5! or 4? (0)
|
||||
# Will stop at 5! or 4? (1)
|
||||
# Will stop at 5! or 4? (2)
|
||||
# Will stop at 5! or 4? (3)
|
||||
# Will stop at 5! or 4? (4)
|
||||
```
|
||||
|
||||
The `range()` function can also modify its 3 defaults arguments. The first two will be the `start` and `stop` values, and the third will be the `step` argument. The step is the amount that the variable is increased by after each iteration.
|
||||
|
||||
```python
|
||||
# range(start, stop, step)
|
||||
>>> for i in range(0, 10, 2):
|
||||
... print(i)
|
||||
...
|
||||
# 0
|
||||
# 2
|
||||
# 4
|
||||
# 6
|
||||
# 8
|
||||
```
|
||||
|
||||
You can even use a negative number for the step argument to make the for loop count down instead of up.
|
||||
|
||||
```python
|
||||
>>> for i in range(5, -1, -1):
|
||||
... print(i)
|
||||
...
|
||||
# 5
|
||||
# 4
|
||||
# 3
|
||||
# 2
|
||||
# 1
|
||||
# 0
|
||||
```
|
||||
|
||||
## For else statement
|
||||
|
||||
This allows to specify a statement to execute in case of the full loop has been executed. Only
|
||||
useful when a `break` condition can occur in the loop:
|
||||
|
||||
```python
|
||||
>>> for i in [1, 2, 3, 4, 5]:
|
||||
... if i == 3:
|
||||
... break
|
||||
... else:
|
||||
... print("only executed when no item is equal to 3")
|
||||
```
|
||||
|
||||
## Ending a Program with sys.exit()
|
||||
|
||||
`exit()` function allows exiting Python.
|
||||
|
||||
```python
|
||||
>>> import sys
|
||||
|
||||
>>> while True:
|
||||
... feedback = input('Type exit to exit: ')
|
||||
... if feedback == 'exit':
|
||||
... print(f'You typed {feedback}.')
|
||||
... sys.exit()
|
||||
...
|
||||
# Type exit to exit: open
|
||||
# Type exit to exit: close
|
||||
# Type exit to exit: exit
|
||||
# You typed exit
|
||||
```
|
||||
@ -1,77 +0,0 @@
|
||||
---
|
||||
title: Python Dataclasses - Python Cheatsheet
|
||||
description: Dataclasses are python classes, but are suited for storing data objects. This module provides a decorator and functions for automatically adding generated special methods such as __init__() and __repr__() to user-defined classes.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Dataclasses
|
||||
</base-title>
|
||||
|
||||
`Dataclasses` are python classes, but are suited for storing data objects.
|
||||
This module provides a decorator and functions for automatically adding generated special methods such as `__init__()` and `__repr__()` to user-defined classes.
|
||||
|
||||
## Features
|
||||
|
||||
1. They store data and represent a certain data type. Ex: A number. For people familiar with ORMs, a model instance is a data object. It represents a specific kind of entity. It holds attributes that define or represent the entity.
|
||||
|
||||
2. They can be compared to other objects of the same type. Ex: A number can be greater than, less than, or equal to another number.
|
||||
|
||||
Python 3.7 provides a decorator dataclass that is used to convert a class into a dataclass.
|
||||
|
||||
```python
|
||||
>>> class Number:
|
||||
... def __init__(self, val):
|
||||
... self.val = val
|
||||
...
|
||||
>>> obj = Number(2)
|
||||
>>> obj.val
|
||||
# 2
|
||||
```
|
||||
|
||||
with dataclass
|
||||
|
||||
```python
|
||||
>>> @dataclass
|
||||
... class Number:
|
||||
... val: int
|
||||
...
|
||||
>>> obj = Number(2)
|
||||
>>> obj.val
|
||||
# 2
|
||||
```
|
||||
|
||||
## Default values
|
||||
|
||||
It is easy to add default values to the fields of your data class.
|
||||
|
||||
```python
|
||||
>>> @dataclass
|
||||
... class Product:
|
||||
... name: str
|
||||
... count: int = 0
|
||||
... price: float = 0.0
|
||||
...
|
||||
>>> obj = Product("Python")
|
||||
>>> obj.name
|
||||
# Python
|
||||
|
||||
>>> obj.count
|
||||
# 0
|
||||
|
||||
>>> obj.price
|
||||
# 0.0
|
||||
```
|
||||
|
||||
## Type hints
|
||||
|
||||
It is mandatory to define the data type in dataclass. However, If you would rather not specify the datatype then, use `typing.Any`.
|
||||
|
||||
```python
|
||||
>>> from dataclasses import dataclass
|
||||
>>> from typing import Any
|
||||
|
||||
>>> @dataclass
|
||||
... class WithoutExplicitTypes:
|
||||
... name: Any
|
||||
... value: Any = 42
|
||||
```
|
||||
@ -1,197 +0,0 @@
|
||||
---
|
||||
title: Python Debugging - Python Cheatsheet
|
||||
description: In computer programming and software development, debugging is the process of finding and resolving bugs (defects or problems that prevent correct operation) within computer programs, software, or systems.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Debugging
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="_blank" href="https://en.wikipedia.org/wiki/Debugging">Finding and resolving bugs</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
In computer programming and software development, debugging is the process of finding and resolving bugs (defects or problems that prevent correct operation) within computer programs, software, or systems.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Raising Exceptions
|
||||
|
||||
Exceptions are raised with a raise statement. In code, a raise statement consists of the following:
|
||||
|
||||
- The `raise` keyword
|
||||
- A call to the `Exception()` function
|
||||
- A string with a helpful error message passed to the `Exception()` function
|
||||
|
||||
```python
|
||||
>>> raise Exception('This is the error message.')
|
||||
# Traceback (most recent call last):
|
||||
# File "<pyshell#191>", line 1, in <module>
|
||||
# raise Exception('This is the error message.')
|
||||
# Exception: This is the error message.
|
||||
```
|
||||
|
||||
Typically, it’s the code that calls the function, not the function itself, that knows how to handle an exception. So, you will commonly see a raise statement inside a function and the `try` and `except` statements in the code calling the function.
|
||||
|
||||
```python
|
||||
>>> def box_print(symbol, width, height):
|
||||
... if len(symbol) != 1:
|
||||
... raise Exception('Symbol must be a single character string.')
|
||||
... if width <= 2:
|
||||
... raise Exception('Width must be greater than 2.')
|
||||
... if height <= 2:
|
||||
... raise Exception('Height must be greater than 2.')
|
||||
... print(symbol * width)
|
||||
... for i in range(height - 2):
|
||||
... print(symbol + (' ' * (width - 2)) + symbol)
|
||||
... print(symbol * width)
|
||||
...
|
||||
>>> for sym, w, h in (('*', 4, 4), ('O', 20, 5), ('x', 1, 3), ('ZZ', 3, 3)):
|
||||
... try:
|
||||
... box_print(sym, w, h)
|
||||
... except Exception as err:
|
||||
... print('An exception happened: ' + str(err))
|
||||
...
|
||||
# ****
|
||||
# * *
|
||||
# * *
|
||||
# ****
|
||||
# OOOOOOOOOOOOOOOOOOOO
|
||||
# O O
|
||||
# O O
|
||||
# O O
|
||||
# OOOOOOOOOOOOOOOOOOOO
|
||||
# An exception happened: Width must be greater than 2.
|
||||
# An exception happened: Symbol must be a single character string.
|
||||
```
|
||||
|
||||
Read more about [Exception Handling](/cheatsheet/exception-handling).
|
||||
|
||||
## Getting the Traceback as a string
|
||||
|
||||
The `traceback` is displayed by Python whenever a raised exception goes unhandled. But can also obtain it as a string by calling traceback.format_exc(). This function is useful if you want the information from an exception’s traceback but also want an except statement to gracefully handle the exception. You will need to import Python’s traceback module before calling this function.
|
||||
|
||||
```python
|
||||
>>> import traceback
|
||||
|
||||
>>> try:
|
||||
... raise Exception('This is the error message.')
|
||||
>>> except:
|
||||
... with open('errorInfo.txt', 'w') as error_file:
|
||||
... error_file.write(traceback.format_exc())
|
||||
... print('The traceback info was written to errorInfo.txt.')
|
||||
...
|
||||
# 116
|
||||
# The traceback info was written to errorInfo.txt.
|
||||
```
|
||||
|
||||
The 116 is the return value from the `write()` method, since 116 characters were written to the file. The `traceback` text was written to errorInfo.txt.
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "<pyshell#28>", line 2, in <module>
|
||||
Exception: This is the error message.
|
||||
|
||||
## Assertions
|
||||
|
||||
An assertion is a sanity check to make sure your code isn’t doing something obviously wrong. These sanity checks are performed by `assert` statements. If the sanity check fails, then an `AssertionError` exception is raised. In code, an `assert` statement consists of the following:
|
||||
|
||||
- The `assert` keyword
|
||||
- A condition (that is, an expression that evaluates to `True` or `False`)
|
||||
- A comma
|
||||
- A `string` to display when the condition is `False`
|
||||
|
||||
```python
|
||||
>>> pod_bay_door_status = 'open'
|
||||
>>> assert pod_bay_door_status == 'open', 'The pod bay doors need to be "open".'
|
||||
|
||||
>>> pod_bay_door_status = 'I\'m sorry, Dave. I\'m afraid I can\'t do that.'
|
||||
>>> assert pod_bay_door_status == 'open', 'The pod bay doors need to be "open".'
|
||||
# Traceback (most recent call last):
|
||||
# File "<pyshell#10>", line 1, in <module>
|
||||
# assert pod_bay_door_status == 'open', 'The pod bay doors need to be "open".'
|
||||
# AssertionError: The pod bay doors need to be "open".
|
||||
```
|
||||
|
||||
In plain English, an assert statement says, “I assert that this condition holds true, and if not, there is a bug somewhere in the program.” Unlike exceptions, your code should not handle assert statements with try and except; if an assert fails, your program should crash. By failing fast like this, you shorten the time between the original cause of the bug and when you first notice the bug. This will reduce the amount of code you will have to check before finding the code that’s causing the bug.
|
||||
|
||||
### Disabling Assertions
|
||||
|
||||
Assertions can be disabled by passing the `-O` option when running Python.
|
||||
|
||||
## Logging
|
||||
|
||||
To enable the `logging` module to display log messages on your screen as your program runs, copy the following to the top of your program:
|
||||
|
||||
```python
|
||||
>>> import logging
|
||||
>>> logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s- %(message)s')
|
||||
```
|
||||
|
||||
Say you wrote a function to calculate the factorial of a number. In mathematics, factorial 4 is 1 × 2 × 3 × 4, or 24. Factorial 7 is 1 × 2 × 3 × 4 × 5 × 6 × 7, or 5,040. Open a new file editor window and enter the following code. It has a bug in it, but you will also enter several log messages to help yourself figure out what is going wrong. Save the program as factorialLog.py.
|
||||
|
||||
```python
|
||||
>>> import logging
|
||||
>>> logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s- %(message)s')
|
||||
>>> logging.debug('Start of program')
|
||||
|
||||
>>> def factorial(n):
|
||||
... logging.debug('Start of factorial(%s)' % (n))
|
||||
... total = 1
|
||||
... for i in range(1, n + 1):
|
||||
... total *= i
|
||||
... logging.debug('i is ' + str(i) + ', total is ' + str(total))
|
||||
... logging.debug('End of factorial(%s)' % (n))
|
||||
... return total
|
||||
...
|
||||
>>> print(factorial(5))
|
||||
>>> logging.debug('End of program')
|
||||
# 2015-05-23 16:20:12,664 - DEBUG - Start of program
|
||||
# 2015-05-23 16:20:12,664 - DEBUG - Start of factorial(5)
|
||||
# 2015-05-23 16:20:12,665 - DEBUG - i is 0, total is 0
|
||||
# 2015-05-23 16:20:12,668 - DEBUG - i is 1, total is 0
|
||||
# 2015-05-23 16:20:12,670 - DEBUG - i is 2, total is 0
|
||||
# 2015-05-23 16:20:12,673 - DEBUG - i is 3, total is 0
|
||||
# 2015-05-23 16:20:12,675 - DEBUG - i is 4, total is 0
|
||||
# 2015-05-23 16:20:12,678 - DEBUG - i is 5, total is 0
|
||||
# 2015-05-23 16:20:12,680 - DEBUG - End of factorial(5)
|
||||
# 0
|
||||
# 2015-05-23 16:20:12,684 - DEBUG - End of program
|
||||
```
|
||||
|
||||
## Logging Levels
|
||||
|
||||
Logging levels provide a way to categorize your log messages by importance. There are five logging levels, described in Table 10-1 from least to most important. Messages can be logged at each level using a different logging function.
|
||||
|
||||
| Level | Logging Function | Description |
|
||||
| ---------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `DEBUG` | `logging.debug()` | The lowest level. Used for small details. Usually you care about these messages only when diagnosing problems. |
|
||||
| `INFO` | `logging.info()` | Used to record information on general events in your program or confirm that things are working at their point in the program. |
|
||||
| `WARNING` | `logging.warning()` | Used to indicate a potential problem that doesn’t prevent the program from working but might do so in the future. |
|
||||
| `ERROR` | `logging.error()` | Used to record an error that caused the program to fail to do something. |
|
||||
| `CRITICAL` | `logging.critical()` | The highest level. Used to indicate a fatal error that has caused or is about to cause the program to stop running entirely. |
|
||||
|
||||
## Disabling Logging
|
||||
|
||||
After you’ve debugged your program, you probably don’t want all these log messages cluttering the screen. The logging.disable() function disables these so that you don’t have to go into your program and remove all the logging calls by hand.
|
||||
|
||||
```python
|
||||
>>> import logging
|
||||
|
||||
>>> logging.basicConfig(level=logging.INFO, format=' %(asctime)s -%(levelname)s - %(message)s')
|
||||
>>> logging.critical('Critical error! Critical error!')
|
||||
# 2015-05-22 11:10:48,054 - CRITICAL - Critical error! Critical error!
|
||||
|
||||
>>> logging.disable(logging.CRITICAL)
|
||||
>>> logging.critical('Critical error! Critical error!')
|
||||
>>> logging.error('Error! Error!')
|
||||
```
|
||||
|
||||
## Logging to a File
|
||||
|
||||
Instead of displaying the log messages to the screen, you can write them to a text file. The `logging.basicConfig()` function takes a filename keyword argument, like so:
|
||||
|
||||
```python
|
||||
>>> import logging
|
||||
>>> logging.basicConfig(filename='myProgramLog.txt', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
```
|
||||
@ -1,188 +0,0 @@
|
||||
---
|
||||
title: Python Decorators - Python Cheatsheet
|
||||
description: A Python Decorator is a syntax that provide a concise and reusable way for extending a function or a class.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Decorators
|
||||
</base-title>
|
||||
|
||||
A Python Decorator provides a concise and reusable way for extending
|
||||
a function or a class.
|
||||
|
||||
## Bare bone decorator
|
||||
|
||||
A decorator in its simplest form is a function that takes another
|
||||
function as an argument and returns a wrapper. The following example
|
||||
shows the creation of a decorator and its usage.
|
||||
|
||||
```python
|
||||
def your_decorator(func):
|
||||
def wrapper():
|
||||
# Do stuff before func...
|
||||
print("Before func!")
|
||||
func()
|
||||
# Do stuff after func...
|
||||
print("After func!")
|
||||
return wrapper
|
||||
|
||||
@your_decorator
|
||||
def foo():
|
||||
print("Hello World!")
|
||||
|
||||
foo()
|
||||
# Before func!
|
||||
# Hello World!
|
||||
# After func!
|
||||
```
|
||||
|
||||
## Decorator for a function with parameters
|
||||
|
||||
```python
|
||||
def your_decorator(func):
|
||||
def wrapper(*args,**kwargs):
|
||||
# Do stuff before func...
|
||||
print("Before func!")
|
||||
func(*args,**kwargs)
|
||||
# Do stuff after func...
|
||||
print("After func!")
|
||||
return wrapper
|
||||
|
||||
@your_decorator
|
||||
def foo(bar):
|
||||
print("My name is " + bar)
|
||||
|
||||
foo("Jack")
|
||||
|
||||
# Before func!
|
||||
# My name is Jack
|
||||
# After func!
|
||||
```
|
||||
|
||||
## Template for a basic decorator
|
||||
|
||||
This template is useful for most decorator use-cases. It is valid for functions
|
||||
with or without parameters, and with or without a return value.
|
||||
|
||||
```python
|
||||
import functools
|
||||
|
||||
def your_decorator(func):
|
||||
@functools.wraps(func) # For preserving the metadata of func.
|
||||
def wrapper(*args,**kwargs):
|
||||
# Do stuff before func...
|
||||
result = func(*args,**kwargs)
|
||||
# Do stuff after func..
|
||||
return result
|
||||
return wrapper
|
||||
```
|
||||
|
||||
## Decorator with parameters
|
||||
|
||||
You can also define parameters for the decorator to use.
|
||||
|
||||
```python
|
||||
import functools
|
||||
|
||||
def your_decorator(arg):
|
||||
def decorator(func):
|
||||
@functools.wraps(func) # For preserving the metadata of func.
|
||||
def wrapper(*args,**kwargs):
|
||||
# Do stuff before func possibly using arg...
|
||||
result = func(*args,**kwargs)
|
||||
# Do stuff after func possibly using arg...
|
||||
return result
|
||||
return wrapper
|
||||
return decorator
|
||||
```
|
||||
|
||||
To use this decorator:
|
||||
|
||||
```python
|
||||
@your_decorator(arg = 'x')
|
||||
def foo(bar):
|
||||
return bar
|
||||
```
|
||||
|
||||
## Class based decorators
|
||||
|
||||
To decorate a class method, you must define the decorator within the class. When
|
||||
only the implicit argument `self` is passed to the method, without any explicit
|
||||
additional arguments, you must make a separate decorator for only those methods
|
||||
without any additional arguments. An example of this, shown below, is when you
|
||||
want to catch and print exceptions in a certain way.
|
||||
|
||||
```python
|
||||
class DecorateMyMethod:
|
||||
|
||||
def decorator_for_class_method_with_no_args(method):
|
||||
def wrapper_for_class_method(self)
|
||||
try:
|
||||
return method(self)
|
||||
except Exception as e:
|
||||
print("\nWARNING: Please make note of the following:\n")
|
||||
print(e)
|
||||
return wrapper_for_class_method
|
||||
|
||||
def __init__(self,succeed:bool):
|
||||
self.succeed = succeed
|
||||
|
||||
@decorator_for_class_method_with_no_args
|
||||
def class_action(self):
|
||||
if self.succeed:
|
||||
print("You succeeded by choice.")
|
||||
else:
|
||||
raise Exception("Epic fail of your own creation.")
|
||||
|
||||
test_succeed = DecorateMyMethods(True)
|
||||
test_succeed.class_action()
|
||||
# You succeeded by choice.
|
||||
|
||||
test_fail = DecorateMyMethod(False)
|
||||
test_fail.class_action()
|
||||
# Exception: Epic fail of your own creation.
|
||||
```
|
||||
|
||||
A decorator can also be defined as a class instead of a method. This is useful
|
||||
for maintaining and updating a state, such as in the following example, where we
|
||||
count the number of calls made to a method:
|
||||
|
||||
```python
|
||||
class CountCallNumber:
|
||||
|
||||
def __init__(self, func):
|
||||
self.func = func
|
||||
self.call_number = 0
|
||||
|
||||
def __call__(self, *args, **kwargs):
|
||||
self.call_number += 1
|
||||
print("This is execution number " + str(self.call_number))
|
||||
return self.func(*args, **kwargs)
|
||||
|
||||
@CountCallNumber
|
||||
def say_hi(name):
|
||||
print("Hi! My name is " + name)
|
||||
|
||||
say_hi("Jack")
|
||||
# This is execution number 1
|
||||
# Hi! My name is Jack
|
||||
|
||||
say_hi("James")
|
||||
# This is execution number 2
|
||||
# Hi! My name is James
|
||||
```
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Count Example
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
This count example is inspired by Patrick Loeber's <a href="https://youtu.be/HGOBQPFzWKo?si=IUvFzeQbzTmeEgKV" target="_blank">YouTube tutorial</a>.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@ -1,269 +0,0 @@
|
||||
---
|
||||
title: Python Dictionaries - Python Cheatsheet
|
||||
description: In Python, a dictionary is an insertion-ordered (from Python > 3.7) collection of key, value pairs.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Dictionaries
|
||||
</base-title>
|
||||
|
||||
In Python, a dictionary is an _ordered_ (from Python > 3.7) collection of `key`: `value` pairs.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the Python 3 <a target="_blank" href="https://docs.python.org/3/tutorial/datastructures.html#dictionaries">documentation</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The main operations on a dictionary are storing a value with some key and extracting the value given the key. It is also possible to delete a key:value pair with <code>del</code>.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Example Dictionary:
|
||||
|
||||
```python
|
||||
my_cat = {
|
||||
'size': 'fat',
|
||||
'color': 'gray',
|
||||
'disposition': 'loud'
|
||||
}
|
||||
```
|
||||
|
||||
## Set key, value using subscript operator `[]`
|
||||
```python
|
||||
>>> my_cat = {
|
||||
... 'size': 'fat',
|
||||
... 'color': 'gray',
|
||||
... 'disposition': 'loud',
|
||||
... }
|
||||
>>> my_cat['age_years'] = 2
|
||||
>>> print(my_cat)
|
||||
...
|
||||
# {'size': 'fat', 'color': 'gray', 'disposition': 'loud', 'age_years': 2}
|
||||
```
|
||||
|
||||
## Get value using subscript operator `[]`
|
||||
|
||||
In case the key is not present in dictionary <a target="_blank" href="https://docs.python.org/3/library/exceptions.html#KeyError">`KeyError`</a> is raised.
|
||||
|
||||
```python
|
||||
>>> my_cat = {
|
||||
... 'size': 'fat',
|
||||
... 'color': 'gray',
|
||||
... 'disposition': 'loud',
|
||||
... }
|
||||
>>> print(my_cat['size'])
|
||||
...
|
||||
# fat
|
||||
>>> print(my_cat['eye_color'])
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# KeyError: 'eye_color'
|
||||
```
|
||||
|
||||
## values()
|
||||
|
||||
The `values()` method gets the **values** of the dictionary:
|
||||
|
||||
```python
|
||||
>>> pet = {'color': 'red', 'age': 42}
|
||||
>>> for value in pet.values():
|
||||
... print(value)
|
||||
...
|
||||
# red
|
||||
# 42
|
||||
```
|
||||
|
||||
## keys()
|
||||
|
||||
The `keys()` method gets the **keys** of the dictionary:
|
||||
|
||||
```python
|
||||
>>> pet = {'color': 'red', 'age': 42}
|
||||
>>> for key in pet.keys():
|
||||
... print(key)
|
||||
...
|
||||
# color
|
||||
# age
|
||||
```
|
||||
|
||||
There is no need to use **.keys()** since by default you will loop through keys:
|
||||
|
||||
```python
|
||||
>>> pet = {'color': 'red', 'age': 42}
|
||||
>>> for key in pet:
|
||||
... print(key)
|
||||
...
|
||||
# color
|
||||
# age
|
||||
```
|
||||
|
||||
## items()
|
||||
|
||||
The `items()` method gets the **items** of a dictionary and returns them as a <router-link to=/cheatsheet/lists-and-tuples#the-tuple-data-type>Tuple</router-link>:
|
||||
|
||||
```python
|
||||
>>> pet = {'color': 'red', 'age': 42}
|
||||
>>> for item in pet.items():
|
||||
... print(item)
|
||||
...
|
||||
# ('color', 'red')
|
||||
# ('age', 42)
|
||||
```
|
||||
|
||||
Using the `keys()`, `values()`, and `items()` methods, a for loop can iterate over the keys, values, or key-value pairs in a dictionary, respectively.
|
||||
|
||||
```python
|
||||
>>> pet = {'color': 'red', 'age': 42}
|
||||
>>> for key, value in pet.items():
|
||||
... print(f'Key: {key} Value: {value}')
|
||||
...
|
||||
# Key: color Value: red
|
||||
# Key: age Value: 42
|
||||
```
|
||||
|
||||
## get()
|
||||
|
||||
The `get()` method returns the value of an item with the given key. If the key doesn't exist, it returns `None`:
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33}
|
||||
|
||||
>>> f'My wife name is {wife.get("name")}'
|
||||
# 'My wife name is Rose'
|
||||
|
||||
>>> f'She is {wife.get("age")} years old.'
|
||||
# 'She is 33 years old.'
|
||||
|
||||
>>> f'She is deeply in love with {wife.get("husband")}'
|
||||
# 'She is deeply in love with None'
|
||||
```
|
||||
|
||||
You can also change the default `None` value to one of your choice:
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33}
|
||||
|
||||
>>> f'She is deeply in love with {wife.get("husband", "lover")}'
|
||||
# 'She is deeply in love with lover'
|
||||
```
|
||||
|
||||
## Adding items with setdefault()
|
||||
|
||||
It's possible to add an item to a dictionary in this way:
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33}
|
||||
>>> if 'has_hair' not in wife:
|
||||
... wife['has_hair'] = True
|
||||
```
|
||||
|
||||
Using the `setdefault` method, we can make the same code more short:
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33}
|
||||
>>> wife.setdefault('has_hair', True)
|
||||
>>> wife
|
||||
# {'name': 'Rose', 'age': 33, 'has_hair': True}
|
||||
```
|
||||
|
||||
## Removing Items
|
||||
|
||||
### pop()
|
||||
|
||||
The `pop()` method removes and returns an item based on a given key.
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33, 'hair': 'brown'}
|
||||
>>> wife.pop('age')
|
||||
# 33
|
||||
>>> wife
|
||||
# {'name': 'Rose', 'hair': 'brown'}
|
||||
```
|
||||
|
||||
### popitem()
|
||||
|
||||
The `popitem()` method removes the last item in a dictionary and returns it.
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33, 'hair': 'brown'}
|
||||
>>> wife.popitem()
|
||||
# ('hair', 'brown')
|
||||
>>> wife
|
||||
# {'name': 'Rose', 'age': 33}
|
||||
```
|
||||
|
||||
### del()
|
||||
|
||||
The `del()` method removes an item based on a given key.
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33, 'hair': 'brown'}
|
||||
>>> del wife['age']
|
||||
>>> wife
|
||||
# {'name': 'Rose', 'hair': 'brown'}
|
||||
```
|
||||
|
||||
### clear()
|
||||
|
||||
The`clear()` method removes all the items in a dictionary.
|
||||
|
||||
```python
|
||||
>>> wife = {'name': 'Rose', 'age': 33, 'hair': 'brown'}
|
||||
>>> wife.clear()
|
||||
>>> wife
|
||||
# {}
|
||||
```
|
||||
|
||||
## Checking keys in a Dictionary
|
||||
|
||||
```python
|
||||
>>> person = {'name': 'Rose', 'age': 33}
|
||||
|
||||
>>> 'name' in person.keys()
|
||||
# True
|
||||
|
||||
>>> 'height' in person.keys()
|
||||
# False
|
||||
|
||||
>>> 'skin' in person # You can omit keys()
|
||||
# False
|
||||
```
|
||||
|
||||
## Checking values in a Dictionary
|
||||
|
||||
```python
|
||||
>>> person = {'name': 'Rose', 'age': 33}
|
||||
|
||||
>>> 'Rose' in person.values()
|
||||
# True
|
||||
|
||||
>>> 33 in person.values()
|
||||
# True
|
||||
```
|
||||
|
||||
## Pretty Printing
|
||||
|
||||
```python
|
||||
>>> import pprint
|
||||
|
||||
>>> wife = {'name': 'Rose', 'age': 33, 'has_hair': True, 'hair_color': 'brown', 'height': 1.6, 'eye_color': 'brown'}
|
||||
>>> pprint.pprint(wife)
|
||||
# {'age': 33,
|
||||
# 'eye_color': 'brown',
|
||||
# 'hair_color': 'brown',
|
||||
# 'has_hair': True,
|
||||
# 'height': 1.6,
|
||||
# 'name': 'Rose'}
|
||||
```
|
||||
|
||||
## Merge two dictionaries
|
||||
|
||||
For Python 3.5+:
|
||||
|
||||
```python
|
||||
>>> dict_a = {'a': 1, 'b': 2}
|
||||
>>> dict_b = {'b': 3, 'c': 4}
|
||||
>>> dict_c = {**dict_a, **dict_b}
|
||||
>>> dict_c
|
||||
# {'a': 1, 'b': 3, 'c': 4}
|
||||
```
|
||||
@ -1,136 +0,0 @@
|
||||
---
|
||||
title: Python Exception Handling - Python Cheatsheet
|
||||
description: In Python, exception handling is the process of responding to the occurrence of exceptions.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Exception Handling
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="_blank" href="https://en.wikipedia.org/wiki/Exception_handling">Exception handling</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
In computing and computer programming, exception handling is the process of responding to the occurrence of exceptions – anomalous or exceptional conditions requiring special processing.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Python has many [built-in exceptions](https://docs.python.org/3/library/exceptions.html) that are raised when a program encounters an error, and most external libraries, like the popular [Requests](https://requests.readthedocs.io/en/latest), include his own [custom exceptions](https://requests.readthedocs.io/en/latest/user/quickstart/#errors-and-exceptions) that we will need to deal to.
|
||||
|
||||
## Basic exception handling
|
||||
|
||||
You can't divide by zero, that is a mathematical true, and if you try to do it in Python, the interpreter will raise the built-in exception [ZeroDivisionError](https://docs.python.org/3/library/exceptions.html#ZeroDivisionError):
|
||||
|
||||
```python
|
||||
>>> def divide(dividend , divisor):
|
||||
... print(dividend / divisor)
|
||||
...
|
||||
>>> divide(dividend=10, divisor=5)
|
||||
# 2
|
||||
|
||||
>>> divide(dividend=10, divisor=0)
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# ZeroDivisionError: division by zero
|
||||
```
|
||||
|
||||
Let's say we don't want our program to stop its execution or show the user an output he will not understand. Say we want to print a useful and clear message, then we need to **_handle_** the exception with the `try` and `except` keywords:
|
||||
|
||||
```python
|
||||
>>> def divide(dividend , divisor):
|
||||
... try:
|
||||
... print(dividend / divisor)
|
||||
... except ZeroDivisionError:
|
||||
... print('You can not divide by 0')
|
||||
...
|
||||
>>> divide(dividend=10, divisor=5)
|
||||
# 2
|
||||
|
||||
>>> divide(dividend=10, divisor=0)
|
||||
# You can not divide by 0
|
||||
```
|
||||
|
||||
## Handling Multiple exceptions using one exception block
|
||||
|
||||
You can also handle multiple exceptions in one line like the following without the need to create multiple exception blocks.
|
||||
|
||||
```python
|
||||
>>> def divide(dividend , divisor):
|
||||
... try:
|
||||
... if (dividend == 10):
|
||||
... var = 'str' + 1
|
||||
... else:
|
||||
... print(dividend / divisor)
|
||||
... except (ZeroDivisionError, TypeError) as error:
|
||||
... print(error)
|
||||
...
|
||||
|
||||
>>> divide(dividend=20, divisor=5)
|
||||
# 4
|
||||
|
||||
>>> divide(dividend=10, divisor=5)
|
||||
# `can only concatenate str (not "int") to str` Error message
|
||||
|
||||
>>> divide(dividend=10, divisor=0)
|
||||
# `division by zero` Error message
|
||||
```
|
||||
|
||||
## Finally code in exception handling
|
||||
|
||||
The code inside the `finally` section is always executed, no matter if an exception has been raised or not:
|
||||
|
||||
```python
|
||||
>>> def divide(dividend , divisor):
|
||||
... try:
|
||||
... print(dividend / divisor)
|
||||
... except ZeroDivisionError:
|
||||
... print('You can not divide by 0')
|
||||
... finally:
|
||||
... print('Execution finished')
|
||||
...
|
||||
>>> divide(dividend=10, divisor=5)
|
||||
# 5
|
||||
# Execution finished
|
||||
|
||||
>>> divide(dividend=10, divisor=0)
|
||||
# You can not divide by 0
|
||||
# Execution finished
|
||||
```
|
||||
|
||||
## Custom Exceptions
|
||||
|
||||
Custom exceptions initialize by creating a `class` that inherits from the base `Exception` class of Python, and are raised using the `raise` keyword:
|
||||
|
||||
```python
|
||||
>>> class MyCustomException(Exception):
|
||||
... pass
|
||||
...
|
||||
>>> raise MyCustomException
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# __main__.MyCustomException
|
||||
```
|
||||
|
||||
To declare a custom exception message, you can pass it as a parameter:
|
||||
|
||||
```python
|
||||
>>> class MyCustomException(Exception):
|
||||
... pass
|
||||
...
|
||||
>>> raise MyCustomException('A custom message for my custom exception')
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# __main__.MyCustomException: A custom message for my custom exception
|
||||
```
|
||||
|
||||
Handling a custom exception is the same as any other:
|
||||
|
||||
```python
|
||||
>>> try:
|
||||
... raise MyCustomException('A custom message for my custom exception')
|
||||
>>> except MyCustomException:
|
||||
... print('My custom exception was raised')
|
||||
...
|
||||
# My custom exception was raised
|
||||
```
|
||||
@ -1,529 +0,0 @@
|
||||
---
|
||||
title: File and directory Paths - Python Cheatsheet
|
||||
description: There are two main modules in Python that deals with path manipulation. One is the os.path module and the other is the pathlib module.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Handling file and directory Paths
|
||||
</base-title>
|
||||
|
||||
There are two main modules in Python that deal with path manipulation.
|
||||
One is the `os.path` module and the other is the `pathlib` module.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
os.path VS pathlib
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The `pathlib` module was added in Python 3.4, offering an object-oriented way to handle file system paths.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Linux and Windows Paths
|
||||
|
||||
On Windows, paths are written using backslashes (`\`) as the separator between
|
||||
folder names. On Unix based operating system such as macOS, Linux, and BSDs,
|
||||
the forward slash (`/`) is used as the path separator. Joining paths can be
|
||||
a headache if your code needs to work on different platforms.
|
||||
|
||||
Fortunately, Python provides easy ways to handle this. We will showcase
|
||||
how to deal with both, `os.path.join` and `pathlib.Path.joinpath`
|
||||
|
||||
Using `os.path.join` on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.path.join('usr', 'bin', 'spam')
|
||||
# 'usr\\bin\\spam'
|
||||
```
|
||||
|
||||
And using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> print(Path('usr').joinpath('bin').joinpath('spam'))
|
||||
# usr/bin/spam
|
||||
```
|
||||
|
||||
`pathlib` also provides a shortcut to joinpath using the `/` operator:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> print(Path('usr') / 'bin' / 'spam')
|
||||
# usr/bin/spam
|
||||
```
|
||||
|
||||
Notice the path separator is different between Windows and Unix based operating
|
||||
system, that's why you want to use one of the above methods instead of
|
||||
adding strings together to join paths together.
|
||||
|
||||
Joining paths is helpful if you need to create different file paths under
|
||||
the same directory.
|
||||
|
||||
Using `os.path.join` on Windows:
|
||||
|
||||
```python
|
||||
>>> my_files = ['accounts.txt', 'details.csv', 'invite.docx']
|
||||
|
||||
>>> for filename in my_files:
|
||||
... print(os.path.join('C:\\Users\\asweigart', filename))
|
||||
...
|
||||
# C:\Users\asweigart\accounts.txt
|
||||
# C:\Users\asweigart\details.csv
|
||||
# C:\Users\asweigart\invite.docx
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> my_files = ['accounts.txt', 'details.csv', 'invite.docx']
|
||||
>>> home = Path.home()
|
||||
>>> for filename in my_files:
|
||||
... print(home / filename)
|
||||
...
|
||||
# /home/asweigart/accounts.txt
|
||||
# /home/asweigart/details.csv
|
||||
# /home/asweigart/invite.docx
|
||||
```
|
||||
|
||||
## The current working directory
|
||||
|
||||
Using `os` on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.getcwd()
|
||||
# 'C:\\Python34'
|
||||
>>> os.chdir('C:\\Windows\\System32')
|
||||
|
||||
>>> os.getcwd()
|
||||
# 'C:\\Windows\\System32'
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
>>> from os import chdir
|
||||
|
||||
>>> print(Path.cwd())
|
||||
# /home/asweigart
|
||||
|
||||
>>> chdir('/usr/lib/python3.6')
|
||||
>>> print(Path.cwd())
|
||||
# /usr/lib/python3.6
|
||||
```
|
||||
|
||||
## Creating new folders
|
||||
|
||||
Using `os` on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>> os.makedirs('C:\\delicious\\walnut\\waffles')
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
>>> cwd = Path.cwd()
|
||||
>>> (cwd / 'delicious' / 'walnut' / 'waffles').mkdir()
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# File "/usr/lib/python3.6/pathlib.py", line 1226, in mkdir
|
||||
# self._accessor.mkdir(self, mode)
|
||||
# File "/usr/lib/python3.6/pathlib.py", line 387, in wrapped
|
||||
# return strfunc(str(pathobj), *args)
|
||||
# FileNotFoundError: [Errno 2] No such file or directory: '/home/asweigart/delicious/walnut/waffles'
|
||||
```
|
||||
|
||||
Oh no, we got a nasty error! The reason is that the 'delicious' directory does
|
||||
not exist, so we cannot make the 'walnut' and the 'waffles' directories under
|
||||
it. To fix this, do:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
>>> cwd = Path.cwd()
|
||||
>>> (cwd / 'delicious' / 'walnut' / 'waffles').mkdir(parents=True)
|
||||
```
|
||||
|
||||
And all is good :)
|
||||
|
||||
## Absolute vs. Relative paths
|
||||
|
||||
There are two ways to specify a file path.
|
||||
|
||||
- An **absolute path**, which always begins with the root folder
|
||||
- A **relative path**, which is relative to the program’s current working directory
|
||||
|
||||
There are also the dot (`.`) and dot-dot (`..`) folders. These are not real folders, but special names that can be used in a path. A single period (“dot”) for a folder name is shorthand for “this directory.” Two periods (“dot-dot”) means “the parent folder.”
|
||||
|
||||
### Handling Absolute paths
|
||||
|
||||
To see if a path is an absolute path:
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>> os.path.isabs('/')
|
||||
# True
|
||||
|
||||
>>> os.path.isabs('..')
|
||||
# False
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
>>> Path('/').is_absolute()
|
||||
# True
|
||||
|
||||
>>> Path('..').is_absolute()
|
||||
# False
|
||||
```
|
||||
|
||||
You can extract an absolute path with both `os.path` and `pathlib`
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>> os.getcwd()
|
||||
'/home/asweigart'
|
||||
|
||||
>>> os.path.abspath('..')
|
||||
'/home'
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
print(Path.cwd())
|
||||
# /home/asweigart
|
||||
|
||||
print(Path('..').resolve())
|
||||
# /home
|
||||
```
|
||||
|
||||
### Handling Relative paths
|
||||
|
||||
You can get a relative path from a starting path to another path.
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>> os.path.relpath('/etc/passwd', '/')
|
||||
# 'etc/passwd'
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
>>> print(Path('/etc/passwd').relative_to('/'))
|
||||
# etc/passwd
|
||||
```
|
||||
|
||||
## Path and File validity
|
||||
|
||||
### Checking if a file/directory exists
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.path.exists('.')
|
||||
# True
|
||||
|
||||
>>> os.path.exists('setup.py')
|
||||
# True
|
||||
|
||||
>>> os.path.exists('/etc')
|
||||
# True
|
||||
|
||||
>>> os.path.exists('nonexistentfile')
|
||||
# False
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
>>> Path('.').exists()
|
||||
# True
|
||||
|
||||
>>> Path('setup.py').exists()
|
||||
# True
|
||||
|
||||
>>> Path('/etc').exists()
|
||||
# True
|
||||
|
||||
>>> Path('nonexistentfile').exists()
|
||||
# False
|
||||
```
|
||||
|
||||
### Checking if a path is a file
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.path.isfile('setup.py')
|
||||
# True
|
||||
|
||||
>>> os.path.isfile('/home')
|
||||
# False
|
||||
|
||||
>>> os.path.isfile('nonexistentfile')
|
||||
# False
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> Path('setup.py').is_file()
|
||||
# True
|
||||
|
||||
>>> Path('/home').is_file()
|
||||
# False
|
||||
|
||||
>>> Path('nonexistentfile').is_file()
|
||||
# False
|
||||
```
|
||||
|
||||
### Checking if a path is a directory
|
||||
|
||||
Using `os.path` on \*nix:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.path.isdir('/')
|
||||
# True
|
||||
|
||||
>>> os.path.isdir('setup.py')
|
||||
# False
|
||||
|
||||
>>> os.path.isdir('/spam')
|
||||
# False
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> Path('/').is_dir()
|
||||
# True
|
||||
|
||||
>>> Path('setup.py').is_dir()
|
||||
# False
|
||||
|
||||
>>> Path('/spam').is_dir()
|
||||
# False
|
||||
```
|
||||
|
||||
## Getting a file's size in bytes
|
||||
|
||||
Using `os.path` on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.path.getsize('C:\\Windows\\System32\\calc.exe')
|
||||
# 776192
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> stat = Path('/bin/python3.6').stat()
|
||||
>>> print(stat) # stat contains some other information about the file as well
|
||||
# os.stat_result(st_mode=33261, st_ino=141087, st_dev=2051, st_nlink=2, st_uid=0,
|
||||
# --snip--
|
||||
# st_gid=0, st_size=10024, st_atime=1517725562, st_mtime=1515119809, st_ctime=1517261276)
|
||||
|
||||
>>> print(stat.st_size) # size in bytes
|
||||
# 10024
|
||||
```
|
||||
|
||||
## Listing directories
|
||||
|
||||
Listing directory contents using `os.listdir` on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
|
||||
>>> os.listdir('C:\\Windows\\System32')
|
||||
# ['0409', '12520437.cpx', '12520850.cpx', '5U877.ax', 'aaclient.dll',
|
||||
# --snip--
|
||||
# 'xwtpdui.dll', 'xwtpw32.dll', 'zh-CN', 'zh-HK', 'zh-TW', 'zipfldr.dll']
|
||||
```
|
||||
|
||||
Listing directory contents using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> for f in Path('/usr/bin').iterdir():
|
||||
... print(f)
|
||||
...
|
||||
# ...
|
||||
# /usr/bin/tiff2rgba
|
||||
# /usr/bin/iconv
|
||||
# /usr/bin/ldd
|
||||
# /usr/bin/cache_restore
|
||||
# /usr/bin/udiskie
|
||||
# /usr/bin/unix2dos
|
||||
# /usr/bin/t1reencode
|
||||
# /usr/bin/epstopdf
|
||||
# /usr/bin/idle3
|
||||
# ...
|
||||
```
|
||||
|
||||
## Directory file sizes
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
WARNING
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
Directories themselves also have a size! So, you might want to check for whether a path is a file or directory using the methods in the methods discussed in the above section.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
Using `os.path.getsize()` and `os.listdir()` together on Windows:
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>> total_size = 0
|
||||
|
||||
>>> for filename in os.listdir('C:\\Windows\\System32'):
|
||||
... total_size = total_size + os.path.getsize(os.path.join('C:\\Windows\\System32', filename))
|
||||
...
|
||||
>>> print(total_size)
|
||||
# 1117846456
|
||||
```
|
||||
|
||||
Using `pathlib` on \*nix:
|
||||
|
||||
```python
|
||||
>>> from pathlib import Path
|
||||
|
||||
>>> total_size = 0
|
||||
>>> for sub_path in Path('/usr/bin').iterdir():
|
||||
... total_size += sub_path.stat().st_size
|
||||
...
|
||||
>>> print(total_size)
|
||||
# 1903178911
|
||||
```
|
||||
|
||||
## Copying files and folders
|
||||
|
||||
The `shutil` module provides functions for copying files, as well as entire folders.
|
||||
|
||||
```python
|
||||
>>> import shutil, os
|
||||
|
||||
>>> os.chdir('C:\\')
|
||||
>>> shutil.copy('C:\\spam.txt', 'C:\\delicious')
|
||||
# C:\\delicious\\spam.txt'
|
||||
|
||||
>>> shutil.copy('eggs.txt', 'C:\\delicious\\eggs2.txt')
|
||||
# 'C:\\delicious\\eggs2.txt'
|
||||
```
|
||||
|
||||
While `shutil.copy()` will copy a single file, `shutil.copytree()` will copy an entire folder and every folder and file contained in it:
|
||||
|
||||
```python
|
||||
>>> import shutil, os
|
||||
|
||||
>>> os.chdir('C:\\')
|
||||
>>> shutil.copytree('C:\\bacon', 'C:\\bacon_backup')
|
||||
# 'C:\\bacon_backup'
|
||||
```
|
||||
|
||||
## Moving and Renaming
|
||||
|
||||
```python
|
||||
>>> import shutil
|
||||
|
||||
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')
|
||||
# 'C:\\eggs\\bacon.txt'
|
||||
```
|
||||
|
||||
The destination path can also specify a filename. In the following example, the source file is moved and renamed:
|
||||
|
||||
```python
|
||||
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs\\new_bacon.txt')
|
||||
# 'C:\\eggs\\new_bacon.txt'
|
||||
```
|
||||
|
||||
If there is no eggs folder, then `move()` will rename bacon.txt to a file named eggs:
|
||||
|
||||
```python
|
||||
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')
|
||||
# 'C:\\eggs'
|
||||
```
|
||||
|
||||
## Deleting files and folders
|
||||
|
||||
- Calling `os.unlink(path)` or `Path.unlink()` will delete the file at path.
|
||||
|
||||
- Calling `os.rmdir(path)` or `Path.rmdir()` will delete the folder at path. This folder must be empty of any files or folders.
|
||||
|
||||
- Calling `shutil.rmtree(path)` will remove the folder at path, and all files and folders it contains will also be deleted.
|
||||
|
||||
## Walking a Directory Tree
|
||||
|
||||
```python
|
||||
>>> import os
|
||||
>>>
|
||||
>>> for folder_name, subfolders, filenames in os.walk('C:\\delicious'):
|
||||
... print(f'The current folder is {folder_name}')
|
||||
... for subfolder in subfolders:
|
||||
... print(f'SUBFOLDER OF {folder_name}: {subfolder}')
|
||||
... for filename in filenames:
|
||||
... print(f'FILE INSIDE {folder_name}: {filename}')
|
||||
... print('')
|
||||
...
|
||||
# The current folder is C:\delicious
|
||||
# SUBFOLDER OF C:\delicious: cats
|
||||
# SUBFOLDER OF C:\delicious: walnut
|
||||
# FILE INSIDE C:\delicious: spam.txt
|
||||
|
||||
# The current folder is C:\delicious\cats
|
||||
# FILE INSIDE C:\delicious\cats: catnames.txt
|
||||
# FILE INSIDE C:\delicious\cats: zophie.jpg
|
||||
|
||||
# The current folder is C:\delicious\walnut
|
||||
# SUBFOLDER OF C:\delicious\walnut: waffles
|
||||
|
||||
# The current folder is C:\delicious\walnut\waffles
|
||||
# FILE INSIDE C:\delicious\walnut\waffles: butter.txt
|
||||
```
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
Pathlib vs Os Module
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
`pathlib` provides a lot more functionality than the ones listed above, like getting file name, getting file extension, reading/writing a file without manually opening it, etc. See the <a target="_blank" href="https://docs.python.org/3/library/pathlib.html">official documentation</a> if you intend to know more.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
@ -1,176 +0,0 @@
|
||||
---
|
||||
title: Python Functions - Python Cheatsheet
|
||||
description: In Python, A function is a block of organized code that is used to perform a single task.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Functions
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="_blank" href="https://en.wikiversity.org/wiki/Programming_Fundamentals/Functions">Programming Functions</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
A function is a block of organized code that is used to perform a single task. They provide better modularity for your application and reuse-ability.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Function Arguments
|
||||
|
||||
A function can take `arguments` and `return values`:
|
||||
|
||||
In the following example, the function **say_hello** receives the argument "name" and prints a greeting:
|
||||
|
||||
```python
|
||||
>>> def say_hello(name):
|
||||
... print(f'Hello {name}')
|
||||
...
|
||||
>>> say_hello('Carlos')
|
||||
# Hello Carlos
|
||||
|
||||
>>> say_hello('Wanda')
|
||||
# Hello Wanda
|
||||
|
||||
>>> say_hello('Rose')
|
||||
# Hello Rose
|
||||
```
|
||||
|
||||
## Keyword Arguments
|
||||
|
||||
To improve code readability, we should be as explicit as possible. We can achieve this in our functions by using `Keyword Arguments`:
|
||||
|
||||
```python
|
||||
>>> def say_hi(name, greeting):
|
||||
... print(f"{greeting} {name}")
|
||||
...
|
||||
>>> # without keyword arguments
|
||||
>>> say_hi('John', 'Hello')
|
||||
# Hello John
|
||||
|
||||
>>> # with keyword arguments
|
||||
>>> say_hi(name='Anna', greeting='Hi')
|
||||
# Hi Anna
|
||||
```
|
||||
|
||||
## Return Values
|
||||
|
||||
When creating a function using the `def` statement, you can specify what the return value should be with a `return` statement. A return statement consists of the following:
|
||||
|
||||
- The `return` keyword.
|
||||
|
||||
- The value or expression that the function should return.
|
||||
|
||||
```python
|
||||
>>> def sum_two_numbers(number_1, number_2):
|
||||
... return number_1 + number_2
|
||||
...
|
||||
>>> result = sum_two_numbers(7, 8)
|
||||
>>> print(result)
|
||||
# 15
|
||||
```
|
||||
|
||||
## Local and Global Scope
|
||||
|
||||
- Code in the global scope cannot use any local variables.
|
||||
|
||||
- However, a local scope can access global variables.
|
||||
|
||||
- Code in a function’s local scope cannot use variables in any other local scope.
|
||||
|
||||
- You can use the same name for different variables if they are in different scopes. That is, there can be a local variable named spam and a global variable also named spam.
|
||||
|
||||
```python
|
||||
global_variable = 'I am available everywhere'
|
||||
|
||||
>>> def some_function():
|
||||
... print(global_variable) # because is global
|
||||
... local_variable = "only available within this function"
|
||||
... print(local_variable)
|
||||
...
|
||||
>>> # the following code will throw error because
|
||||
>>> # 'local_variable' only exists inside 'some_function'
|
||||
>>> print(local_variable)
|
||||
Traceback (most recent call last):
|
||||
File "<stdin>", line 10, in <module>
|
||||
NameError: name 'local_variable' is not defined
|
||||
```
|
||||
|
||||
## The global Statement
|
||||
|
||||
If you need to modify a global variable from within a function, use the global statement:
|
||||
|
||||
```python
|
||||
>>> def spam():
|
||||
... global eggs
|
||||
... eggs = 'spam'
|
||||
...
|
||||
>>> eggs = 'global'
|
||||
>>> spam()
|
||||
>>> print(eggs)
|
||||
```
|
||||
|
||||
There are four rules to tell whether a variable is in a local scope or global scope:
|
||||
|
||||
1. If a variable is being used in the global scope (that is, outside all functions), then it is always a global variable.
|
||||
|
||||
1. If there is a global statement for that variable in a function, it is a global variable.
|
||||
|
||||
1. Otherwise, if the variable is used in an assignment statement in the function, it is a local variable.
|
||||
|
||||
1. But if the variable is not used in an assignment statement, it is a global variable.
|
||||
|
||||
## Lambda Functions
|
||||
|
||||
In Python, a lambda function is a single-line, anonymous function, which can have any number of arguments, but it can only have one expression.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the <a target="_blank" href="https://docs.python.org/3/library/ast.html?highlight=lambda#function-and-class-definitions">Python 3 Tutorial</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
lambda is a minimal function definition that can be used inside an expression. Unlike FunctionDef, body holds a single node.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
Single line expression
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
Lambda functions can only evaluate an expression, like a single line of code.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
This function:
|
||||
|
||||
```python
|
||||
>>> def add(x, y):
|
||||
... return x + y
|
||||
...
|
||||
>>> add(5, 3)
|
||||
# 8
|
||||
```
|
||||
|
||||
Is equivalent to the _lambda_ function:
|
||||
|
||||
```python
|
||||
>>> add = lambda x, y: x + y
|
||||
>>> add(5, 3)
|
||||
# 8
|
||||
```
|
||||
|
||||
Like regular nested functions, lambdas also work as lexical closures:
|
||||
|
||||
```python
|
||||
>>> def make_adder(n):
|
||||
... return lambda x: x + n
|
||||
...
|
||||
>>> plus_3 = make_adder(3)
|
||||
>>> plus_5 = make_adder(5)
|
||||
|
||||
>>> plus_3(4)
|
||||
# 7
|
||||
>>> plus_5(4)
|
||||
# 9
|
||||
```
|
||||
@ -1,69 +0,0 @@
|
||||
---
|
||||
title: Python Json and YAML - Python Cheatsheet
|
||||
description: JSON stands for JavaScript Object Notation and is a lightweight format for storing and transporting data. Json is often used when data is sent from a server to a web page.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
JSON and YAML
|
||||
</base-title>
|
||||
|
||||
## JSON
|
||||
|
||||
JSON stands for JavaScript Object Notation and is a lightweight format for storing and transporting data. Json is often used when data is sent from a server to a web page.
|
||||
|
||||
```python
|
||||
>>> import json
|
||||
>>> with open("filename.json", "r") as f:
|
||||
... content = json.load(f)
|
||||
```
|
||||
|
||||
Write a JSON file with:
|
||||
|
||||
```python
|
||||
>>> import json
|
||||
|
||||
>>> content = {"name": "Joe", "age": 20}
|
||||
>>> with open("filename.json", "w") as f:
|
||||
... json.dump(content, f, indent=2)
|
||||
```
|
||||
|
||||
## YAML
|
||||
|
||||
Compared to JSON, YAML allows a much better human maintainability and gives ability to add comments. It is a convenient choice for configuration files where a human will have to edit.
|
||||
|
||||
There are two main libraries allowing access to YAML files:
|
||||
|
||||
- [PyYaml](https://pypi.python.org/pypi/PyYAML)
|
||||
- [Ruamel.yaml](https://pypi.python.org/pypi/ruamel.yaml)
|
||||
|
||||
Install them using `pip install` in your virtual environment.
|
||||
|
||||
The first one is easier to use but the second one, Ruamel, implements much better the YAML
|
||||
specification, and allow for example to modify a YAML content without altering comments.
|
||||
|
||||
Open a YAML file with:
|
||||
|
||||
```python
|
||||
>>> from ruamel.yaml import YAML
|
||||
|
||||
>>> with open("filename.yaml") as f:
|
||||
... yaml=YAML()
|
||||
... yaml.load(f)
|
||||
```
|
||||
|
||||
## Anyconfig
|
||||
|
||||
[Anyconfig](https://pypi.python.org/pypi/anyconfig) is a very handy package, allowing to abstract completely the underlying configuration file format. It allows to load a Python dictionary from JSON, YAML, TOML, and so on.
|
||||
|
||||
Install it with:
|
||||
|
||||
```bash
|
||||
pip install anyconfig
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```python
|
||||
>>> import anyconfig
|
||||
>>> conf1 = anyconfig.load("/path/to/foo/conf.d/a.yml")
|
||||
```
|
||||
@ -1,394 +0,0 @@
|
||||
---
|
||||
title: Python Lists and Tuples - Python Cheatsheet
|
||||
description: In python, Lists are are one of the 4 data types in Python used to store collections of data.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Lists
|
||||
</base-title>
|
||||
|
||||
Lists are one of the 4 data types in Python used to store collections of data.
|
||||
|
||||
```python
|
||||
['John', 'Peter', 'Debora', 'Charles']
|
||||
```
|
||||
|
||||
## Getting values with indexes
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[0]
|
||||
# 'table'
|
||||
|
||||
>>> furniture[1]
|
||||
# 'chair'
|
||||
|
||||
>>> furniture[2]
|
||||
# 'rack'
|
||||
|
||||
>>> furniture[3]
|
||||
# 'shelf'
|
||||
```
|
||||
|
||||
## Negative indexes
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[-1]
|
||||
# 'shelf'
|
||||
|
||||
>>> furniture[-3]
|
||||
# 'chair'
|
||||
|
||||
>>> f'The {furniture[-1]} is bigger than the {furniture[-3]}'
|
||||
# 'The shelf is bigger than the chair'
|
||||
```
|
||||
|
||||
## Getting sublists with Slices
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[0:4]
|
||||
# ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[1:3]
|
||||
# ['chair', 'rack']
|
||||
|
||||
>>> furniture[0:-1]
|
||||
# ['table', 'chair', 'rack']
|
||||
|
||||
>>> furniture[:2]
|
||||
# ['table', 'chair']
|
||||
|
||||
>>> furniture[1:]
|
||||
# ['chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[:]
|
||||
# ['table', 'chair', 'rack', 'shelf']
|
||||
```
|
||||
|
||||
Slicing the complete list will perform a copy:
|
||||
|
||||
```python
|
||||
>>> spam2 = spam[:]
|
||||
# ['cat', 'bat', 'rat', 'elephant']
|
||||
|
||||
>>> spam.append('dog')
|
||||
>>> spam
|
||||
# ['cat', 'bat', 'rat', 'elephant', 'dog']
|
||||
|
||||
>>> spam2
|
||||
# ['cat', 'bat', 'rat', 'elephant']
|
||||
```
|
||||
|
||||
## Getting a list length with len()
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> len(furniture)
|
||||
# 4
|
||||
```
|
||||
|
||||
## Changing values with indexes
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[0] = 'desk'
|
||||
>>> furniture
|
||||
# ['desk', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> furniture[2] = furniture[1]
|
||||
>>> furniture
|
||||
# ['desk', 'chair', 'chair', 'shelf']
|
||||
|
||||
>>> furniture[-1] = 'bed'
|
||||
>>> furniture
|
||||
# ['desk', 'chair', 'chair', 'bed']
|
||||
```
|
||||
|
||||
## Concatenation and Replication
|
||||
|
||||
```python
|
||||
>>> [1, 2, 3] + ['A', 'B', 'C']
|
||||
# [1, 2, 3, 'A', 'B', 'C']
|
||||
|
||||
>>> ['X', 'Y', 'Z'] * 3
|
||||
# ['X', 'Y', 'Z', 'X', 'Y', 'Z', 'X', 'Y', 'Z']
|
||||
|
||||
>>> my_list = [1, 2, 3]
|
||||
>>> my_list = my_list + ['A', 'B', 'C']
|
||||
>>> my_list
|
||||
# [1, 2, 3, 'A', 'B', 'C']
|
||||
```
|
||||
|
||||
## Using for loops with Lists
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> for item in furniture:
|
||||
... print(item)
|
||||
# table
|
||||
# chair
|
||||
# rack
|
||||
# shelf
|
||||
```
|
||||
|
||||
## Getting the index in a loop with enumerate()
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
|
||||
>>> for index, item in enumerate(furniture):
|
||||
... print(f'index: {index} - item: {item}')
|
||||
# index: 0 - item: table
|
||||
# index: 1 - item: chair
|
||||
# index: 2 - item: rack
|
||||
# index: 3 - item: shelf
|
||||
```
|
||||
|
||||
## Loop in Multiple Lists with zip()
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> price = [100, 50, 80, 40]
|
||||
|
||||
>>> for item, amount in zip(furniture, price):
|
||||
... print(f'The {item} costs ${amount}')
|
||||
# The table costs $100
|
||||
# The chair costs $50
|
||||
# The rack costs $80
|
||||
# The shelf costs $40
|
||||
```
|
||||
|
||||
## The in and not in operators
|
||||
|
||||
```python
|
||||
>>> 'rack' in ['table', 'chair', 'rack', 'shelf']
|
||||
# True
|
||||
|
||||
>>> 'bed' in ['table', 'chair', 'rack', 'shelf']
|
||||
# False
|
||||
|
||||
>>> 'bed' not in furniture
|
||||
# True
|
||||
|
||||
>>> 'rack' not in furniture
|
||||
# False
|
||||
```
|
||||
|
||||
## The Multiple Assignment Trick
|
||||
|
||||
The multiple assignment trick is a shortcut that lets you assign multiple variables with the values in a list in one line of code. So instead of doing this:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> table = furniture[0]
|
||||
>>> chair = furniture[1]
|
||||
>>> rack = furniture[2]
|
||||
>>> shelf = furniture[3]
|
||||
```
|
||||
|
||||
You could type this line of code:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> table, chair, rack, shelf = furniture
|
||||
|
||||
>>> table
|
||||
# 'table'
|
||||
|
||||
>>> chair
|
||||
# 'chair'
|
||||
|
||||
>>> rack
|
||||
# 'rack'
|
||||
|
||||
>>> shelf
|
||||
# 'shelf'
|
||||
```
|
||||
|
||||
The multiple assignment trick can also be used to swap the values in two variables:
|
||||
|
||||
```python
|
||||
>>> a, b = 'table', 'chair'
|
||||
>>> a, b = b, a
|
||||
>>> print(a)
|
||||
# chair
|
||||
|
||||
>>> print(b)
|
||||
# table
|
||||
```
|
||||
|
||||
## The index Method
|
||||
|
||||
The `index` method allows you to find the index of a value by passing its name:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> furniture.index('chair')
|
||||
# 1
|
||||
```
|
||||
|
||||
## Adding Values
|
||||
|
||||
### append()
|
||||
|
||||
`append` adds an element to the end of a `list`:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> furniture.append('bed')
|
||||
>>> furniture
|
||||
# ['table', 'chair', 'rack', 'shelf', 'bed']
|
||||
```
|
||||
|
||||
### insert()
|
||||
|
||||
`insert` adds an element to a `list` at a given position:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> furniture.insert(1, 'bed')
|
||||
>>> furniture
|
||||
# ['table', 'bed', 'chair', 'rack', 'shelf']
|
||||
```
|
||||
|
||||
## Removing Values
|
||||
|
||||
### del()
|
||||
|
||||
`del` removes an item using the index:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> del furniture[2]
|
||||
>>> furniture
|
||||
# ['table', 'chair', 'shelf']
|
||||
|
||||
>>> del furniture[2]
|
||||
>>> furniture
|
||||
# ['table', 'chair']
|
||||
```
|
||||
|
||||
### remove()
|
||||
|
||||
`remove` removes an item with using actual value of it:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> furniture.remove('chair')
|
||||
>>> furniture
|
||||
# ['table', 'rack', 'shelf']
|
||||
```
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
Removing repeated items
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
If the value appears multiple times in the list, only the first instance of the value will be removed.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
### pop()
|
||||
|
||||
By default, `pop` will remove and return the last item of the list. You can also pass the index of the element as an optional parameter:
|
||||
|
||||
```python
|
||||
>>> animals = ['cat', 'bat', 'rat', 'elephant']
|
||||
|
||||
>>> animals.pop()
|
||||
'elephant'
|
||||
|
||||
>>> animals
|
||||
['cat', 'bat', 'rat']
|
||||
|
||||
>>> animals.pop(0)
|
||||
'cat'
|
||||
|
||||
>>> animals
|
||||
['bat', 'rat']
|
||||
```
|
||||
|
||||
## Sorting values with sort()
|
||||
|
||||
```python
|
||||
>>> numbers = [2, 5, 3.14, 1, -7]
|
||||
>>> numbers.sort()
|
||||
>>> numbers
|
||||
# [-7, 1, 2, 3.14, 5]
|
||||
|
||||
furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
furniture.sort()
|
||||
furniture
|
||||
# ['chair', 'rack', 'shelf', 'table']
|
||||
```
|
||||
|
||||
You can also pass `True` for the `reverse` keyword argument to have `sort()` sort the values in reverse order:
|
||||
|
||||
```python
|
||||
>>> furniture.sort(reverse=True)
|
||||
>>> furniture
|
||||
# ['table', 'shelf', 'rack', 'chair']
|
||||
```
|
||||
|
||||
If you need to sort the values in regular alphabetical order, pass `str.lower` for the key keyword argument in the sort() method call:
|
||||
|
||||
```python
|
||||
>>> letters = ['a', 'z', 'A', 'Z']
|
||||
>>> letters.sort(key=str.lower)
|
||||
>>> letters
|
||||
# ['a', 'A', 'z', 'Z']
|
||||
```
|
||||
|
||||
You can use the built-in function `sorted` to return a new list:
|
||||
|
||||
```python
|
||||
>>> furniture = ['table', 'chair', 'rack', 'shelf']
|
||||
>>> sorted(furniture)
|
||||
# ['chair', 'rack', 'shelf', 'table']
|
||||
```
|
||||
|
||||
## The Tuple data type
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="_blank" href="https://stackoverflow.com/questions/1708510/list-vs-tuple-when-to-use-each">Tuples vs Lists</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The key difference between tuples and lists is that, while <code>tuples</code> are <i>immutable</i> objects, <code>lists</code> are <i>mutable</i>. This means that tuples cannot be changed while the lists can be modified. Tuples are more memory efficient than the lists.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
```python
|
||||
>>> furniture = ('table', 'chair', 'rack', 'shelf')
|
||||
|
||||
>>> furniture[0]
|
||||
# 'table'
|
||||
|
||||
>>> furniture[1:3]
|
||||
# ('chair', 'rack')
|
||||
|
||||
>>> len(furniture)
|
||||
# 4
|
||||
```
|
||||
|
||||
The main way that tuples are different from lists is that tuples, like strings, are immutable.
|
||||
|
||||
## Converting between list() and tuple()
|
||||
|
||||
```python
|
||||
>>> tuple(['cat', 'dog', 5])
|
||||
# ('cat', 'dog', 5)
|
||||
|
||||
>>> list(('cat', 'dog', 5))
|
||||
# ['cat', 'dog', 5]
|
||||
|
||||
>>> list('hello')
|
||||
# ['h', 'e', 'l', 'l', 'o']
|
||||
```
|
||||
@ -1,40 +0,0 @@
|
||||
---
|
||||
title: Python Main function - Python Cheatsheet
|
||||
description: is the name of the scope in which top-level code executes. A module’s name is set equal to main when read from standard input, a script, or from an interactive prompt.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Main top-level script environment
|
||||
</base-title>
|
||||
|
||||
## What is it
|
||||
|
||||
`__main__` is the name of the scope in which top-level code executes.
|
||||
A module’s **name** is set equal to `__main__` when read from standard input, a script, or from an interactive prompt.
|
||||
|
||||
A module can discover whether it is running in the main scope by checking its own `__name__`, which allows a common idiom for conditionally executing code in a module. When it is run as a script or with `python -m` but not when it is imported:
|
||||
|
||||
```python
|
||||
>>> if __name__ == "__main__":
|
||||
... # execute only if run as a script
|
||||
... main()
|
||||
```
|
||||
|
||||
For a package, the same effect can be achieved by including a **main**.py module, the contents of which will be executed when the module is run with -m.
|
||||
|
||||
For example, we are developing a script designed to be used as a module, we should do:
|
||||
|
||||
```python
|
||||
>>> def add(a, b):
|
||||
... return a+b
|
||||
...
|
||||
>>> if __name__ == "__main__":
|
||||
... add(3, 5)
|
||||
```
|
||||
|
||||
## Advantages
|
||||
|
||||
1. Every Python module has it’s `__name__` defined and if this is `__main__`, it implies that the module is run standalone by the user, and we can do corresponding appropriate actions.
|
||||
2. If you import this script as a module in another script, the **name** is set to the name of the script/module.
|
||||
3. Python files can act as either reusable modules, or as standalone programs.
|
||||
4. `if __name__ == "__main__":` is used to execute some code only if the file is run directly, and is not being imported.
|
||||
@ -1,332 +0,0 @@
|
||||
---
|
||||
title: Manipulating strings - Python Cheatsheet
|
||||
description: An escape character is created by typing a backslash \ followed by the character you want to insert.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Manipulating Strings
|
||||
</base-title>
|
||||
|
||||
## Escape characters
|
||||
|
||||
An escape character is created by typing a backslash `\` followed by the character you want to insert.
|
||||
|
||||
| Escape character | Prints as |
|
||||
| ---------------- | -------------------- |
|
||||
| `\'` | Single quote |
|
||||
| `\"` | Double quote |
|
||||
| `\t` | Tab |
|
||||
| `\n` | Newline (line break) |
|
||||
| `\\` | Backslash |
|
||||
| `\b` | Backspace |
|
||||
| `\ooo` | Octal value |
|
||||
| `\r` | Carriage Return |
|
||||
|
||||
```python
|
||||
>>> print("Hello there!\nHow are you?\nI\'m doing fine.")
|
||||
# Hello there!
|
||||
# How are you?
|
||||
# I'm doing fine.
|
||||
```
|
||||
|
||||
## Raw strings
|
||||
|
||||
A raw string entirely ignores all escape characters and prints any backslash that appears in the string.
|
||||
|
||||
```python
|
||||
>>> print(r"Hello there!\nHow are you?\nI\'m doing fine.")
|
||||
# Hello there!\nHow are you?\nI\'m doing fine.
|
||||
```
|
||||
|
||||
Raw strings are mostly used for <router-link to="/cheatsheet/regular-expressions">regular expression</router-link> definition.
|
||||
|
||||
## Multiline Strings
|
||||
|
||||
```python
|
||||
>>> print(
|
||||
... """Dear Alice,
|
||||
...
|
||||
... Eve's cat has been arrested for catnapping,
|
||||
... cat burglary, and extortion.
|
||||
...
|
||||
... Sincerely,
|
||||
... Bob"""
|
||||
... )
|
||||
|
||||
# Dear Alice,
|
||||
|
||||
# Eve's cat has been arrested for catnapping,
|
||||
# cat burglary, and extortion.
|
||||
|
||||
# Sincerely,
|
||||
# Bob
|
||||
```
|
||||
|
||||
## Indexing and Slicing strings
|
||||
|
||||
H e l l o w o r l d !
|
||||
0 1 2 3 4 5 6 7 8 9 10 11
|
||||
|
||||
### Indexing
|
||||
|
||||
```python
|
||||
>>> spam = 'Hello world!'
|
||||
|
||||
>>> spam[0]
|
||||
# 'H'
|
||||
|
||||
>>> spam[4]
|
||||
# 'o'
|
||||
|
||||
>>> spam[-1]
|
||||
# '!'
|
||||
```
|
||||
|
||||
### Slicing
|
||||
|
||||
```python
|
||||
>>> spam = 'Hello world!'
|
||||
|
||||
>>> spam[0:5]
|
||||
# 'Hello'
|
||||
|
||||
>>> spam[:5]
|
||||
# 'Hello'
|
||||
|
||||
>>> spam[6:]
|
||||
# 'world!'
|
||||
|
||||
>>> spam[6:-1]
|
||||
# 'world'
|
||||
|
||||
>>> spam[:-1]
|
||||
# 'Hello world'
|
||||
|
||||
>>> spam[::-1]
|
||||
# '!dlrow olleH'
|
||||
|
||||
>>> fizz = spam[0:5]
|
||||
>>> fizz
|
||||
# 'Hello'
|
||||
```
|
||||
|
||||
## The in and not in operators
|
||||
|
||||
```python
|
||||
>>> 'Hello' in 'Hello World'
|
||||
# True
|
||||
|
||||
>>> 'Hello' in 'Hello'
|
||||
# True
|
||||
|
||||
>>> 'HELLO' in 'Hello World'
|
||||
# False
|
||||
|
||||
>>> '' in 'spam'
|
||||
# True
|
||||
|
||||
>>> 'cats' not in 'cats and dogs'
|
||||
# False
|
||||
```
|
||||
|
||||
## upper(), lower() and title()
|
||||
|
||||
Transforms a string to upper, lower and title case:
|
||||
|
||||
```python
|
||||
>>> greet = 'Hello world!'
|
||||
>>> greet.upper()
|
||||
# 'HELLO WORLD!'
|
||||
|
||||
>>> greet.lower()
|
||||
# 'hello world!'
|
||||
|
||||
>>> greet.title()
|
||||
# 'Hello World!'
|
||||
```
|
||||
|
||||
## isupper() and islower() methods
|
||||
|
||||
Returns `True` or `False` after evaluating if a string is in upper or lower case:
|
||||
|
||||
```python
|
||||
>>> spam = 'Hello world!'
|
||||
>>> spam.islower()
|
||||
# False
|
||||
|
||||
>>> spam.isupper()
|
||||
# False
|
||||
|
||||
>>> 'HELLO'.isupper()
|
||||
# True
|
||||
|
||||
>>> 'abc12345'.islower()
|
||||
# True
|
||||
|
||||
>>> '12345'.islower()
|
||||
# False
|
||||
|
||||
>>> '12345'.isupper()
|
||||
# False
|
||||
```
|
||||
|
||||
## The isX string methods
|
||||
|
||||
| Method | Description |
|
||||
| ----------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| isalpha() | returns `True` if the string consists only of letters. |
|
||||
| isalnum() | returns `True` if the string consists only of letters and numbers. |
|
||||
| isdecimal() | returns `True` if the string consists only of numbers. |
|
||||
| isspace() | returns `True` if the string consists only of spaces, tabs, and new-lines. |
|
||||
| istitle() | returns `True` if the string consists only of words that begin with an uppercase letter followed by only lowercase characters. |
|
||||
|
||||
## startswith() and endswith()
|
||||
|
||||
```python
|
||||
>>> 'Hello world!'.startswith('Hello')
|
||||
# True
|
||||
|
||||
>>> 'Hello world!'.endswith('world!')
|
||||
# True
|
||||
|
||||
>>> 'abc123'.startswith('abcdef')
|
||||
# False
|
||||
|
||||
>>> 'abc123'.endswith('12')
|
||||
# False
|
||||
|
||||
>>> 'Hello world!'.startswith('Hello world!')
|
||||
# True
|
||||
|
||||
>>> 'Hello world!'.endswith('Hello world!')
|
||||
# True
|
||||
```
|
||||
|
||||
## join() and split()
|
||||
|
||||
### join()
|
||||
|
||||
The `join()` method takes all the items in an iterable, like a <router-link to="/cheatsheet/lists-and-tuples">list</router-link>, <router-link to="/cheatsheet/dictionaries">dictionary</router-link>, <router-link to="/cheatsheet/lists-and-tuples#the-tuple-data-type">tuple</router-link> or <router-link to="/cheatsheet/sets">set</router-link>, and joins them into a string. You can also specify a separator.
|
||||
|
||||
```python
|
||||
>>> ''.join(['My', 'name', 'is', 'Simon'])
|
||||
'MynameisSimon'
|
||||
|
||||
>>> ', '.join(['cats', 'rats', 'bats'])
|
||||
# 'cats, rats, bats'
|
||||
|
||||
>>> ' '.join(['My', 'name', 'is', 'Simon'])
|
||||
# 'My name is Simon'
|
||||
|
||||
>>> 'ABC'.join(['My', 'name', 'is', 'Simon'])
|
||||
# 'MyABCnameABCisABCSimon'
|
||||
```
|
||||
|
||||
### split()
|
||||
|
||||
The `split()` method splits a `string` into a `list`. By default, it will use whitespace to separate the items, but you can also set another character of choice:
|
||||
|
||||
```python
|
||||
>>> 'My name is Simon'.split()
|
||||
# ['My', 'name', 'is', 'Simon']
|
||||
|
||||
>>> 'MyABCnameABCisABCSimon'.split('ABC')
|
||||
# ['My', 'name', 'is', 'Simon']
|
||||
|
||||
>>> 'My name is Simon'.split('m')
|
||||
# ['My na', 'e is Si', 'on']
|
||||
|
||||
>>> ' My name is Simon'.split()
|
||||
# ['My', 'name', 'is', 'Simon']
|
||||
|
||||
>>> ' My name is Simon'.split(' ')
|
||||
# ['', 'My', '', 'name', 'is', '', 'Simon']
|
||||
```
|
||||
|
||||
## Justifying text with rjust(), ljust() and center()
|
||||
|
||||
```python
|
||||
>>> 'Hello'.rjust(10)
|
||||
# ' Hello'
|
||||
|
||||
>>> 'Hello'.rjust(20)
|
||||
# ' Hello'
|
||||
|
||||
>>> 'Hello World'.rjust(20)
|
||||
# ' Hello World'
|
||||
|
||||
>>> 'Hello'.ljust(10)
|
||||
# 'Hello '
|
||||
|
||||
>>> 'Hello'.center(20)
|
||||
# ' Hello '
|
||||
```
|
||||
|
||||
An optional second argument to `rjust()` and `ljust()` will specify a fill character apart from a space character:
|
||||
|
||||
```python
|
||||
>>> 'Hello'.rjust(20, '*')
|
||||
# '***************Hello'
|
||||
|
||||
>>> 'Hello'.ljust(20, '-')
|
||||
# 'Hello---------------'
|
||||
|
||||
>>> 'Hello'.center(20, '=')
|
||||
# '=======Hello========'
|
||||
```
|
||||
|
||||
## Removing whitespace with strip(), rstrip(), and lstrip()
|
||||
|
||||
```python
|
||||
>>> spam = ' Hello World '
|
||||
>>> spam.strip()
|
||||
# 'Hello World'
|
||||
|
||||
>>> spam.lstrip()
|
||||
# 'Hello World '
|
||||
|
||||
>>> spam.rstrip()
|
||||
# ' Hello World'
|
||||
|
||||
>>> spam = 'SpamSpamBaconSpamEggsSpamSpam'
|
||||
>>> spam.strip('ampS')
|
||||
# 'BaconSpamEggs'
|
||||
```
|
||||
|
||||
## The Count Method
|
||||
|
||||
Counts the number of occurrences of a given character or substring in the string it is applied to. Can be optionally provided start and end index.
|
||||
|
||||
```python
|
||||
>>> sentence = 'one sheep two sheep three sheep four'
|
||||
>>> sentence.count('sheep')
|
||||
# 3
|
||||
|
||||
>>> sentence.count('e')
|
||||
# 9
|
||||
|
||||
>>> sentence.count('e', 6)
|
||||
# 8
|
||||
# returns count of e after 'one sh' i.e 6 chars since beginning of string
|
||||
|
||||
>>> sentence.count('e', 7)
|
||||
# 7
|
||||
```
|
||||
|
||||
## Replace Method
|
||||
|
||||
Replaces all occurences of a given substring with another substring. Can be optionally provided a third argument to limit the number of replacements. Returns a new string.
|
||||
|
||||
```python
|
||||
>>> text = "Hello, world!"
|
||||
>>> text.replace("world", "planet")
|
||||
# 'Hello, planet!'
|
||||
|
||||
>>> fruits = "apple, banana, cherry, apple"
|
||||
>>> fruits.replace("apple", "orange", 1)
|
||||
# 'orange, banana, cherry, apple'
|
||||
|
||||
>>> sentence = "I like apples, Apples are my favorite fruit"
|
||||
>>> sentence.replace("apples", "oranges")
|
||||
# 'I like oranges, Apples are my favorite fruit'
|
||||
```
|
||||
@ -1,216 +0,0 @@
|
||||
---
|
||||
title: Python OOP Basics - Python Cheatsheet
|
||||
|
||||
description: Object-Oriented Programming (OOP) is a programming paradigm that revolves around the concept of objects, which are instances of classes. OOP principles are fundamental concepts that guide the design and development of software in an object-oriented way. In Python, OOP is supported by the use of classes and objects. Here are some of the basic OOP principles in Python
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python OOP Basics
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a href="https://en.wikipedia.org/wiki/Object-oriented_programming">Object-Oriented Programming</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Object-oriented programming (OOP) is a programming paradigm based on the concept of "objects", which can contain data and code. The data is in the form of fields (often known as attributes or properties), and the code is in the form of procedures (often known as methods).
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## Encapsulation
|
||||
|
||||
Encapsulation is one of the fundamental concepts of object-oriented programming, which helps to protect the data and methods of an object from unauthorized access and modification. It is a way to achieve data abstraction, which means that the implementation details of an object are hidden from the outside world, and only the essential information is exposed.
|
||||
|
||||
In Python, encapsulation can be achieved by using access modifiers. Access modifiers are keywords that define the accessibility of attributes and methods in a class. The three access modifiers available in Python are public, private, and protected. However, Python does not have an explicit way of defining access modifiers like some other programming languages such as Java and C++. Instead, it uses a convention of using underscore prefixes to indicate the access level.
|
||||
|
||||
In the given code example, the class MyClass has two attributes, _protected_var and __private_var. The _protected_var is marked as protected by using a single underscore prefix. This means that the attribute can be accessed within the class and its subclasses but not outside the class. The __private_var is marked as private by using two underscore prefixes. This means that the attribute can only be accessed within the class and not outside the class, not even in its subclasses.
|
||||
|
||||
When we create an object of the MyClass class, we can access the _protected_var attribute using the object name with a single underscore prefix. However, we cannot access the __private_var attribute using the object name, as it is hidden from the outside world. If we try to access the __private_var attribute, we will get an AttributeError as shown in the code.
|
||||
|
||||
In summary, encapsulation is an important concept in object-oriented programming that helps to protect the implementation details of an object. In Python, we can achieve encapsulation by using access modifiers and using underscore prefixes to indicate the access level.
|
||||
|
||||
```python
|
||||
# Define a class named MyClass
|
||||
class MyClass:
|
||||
|
||||
# Constructor method that initializes the class object
|
||||
def __init__(self):
|
||||
|
||||
# Define a protected variable with an initial value of 10
|
||||
# The variable name starts with a single underscore, which indicates protected access
|
||||
self._protected_var = 10
|
||||
|
||||
# Define a private variable with an initial value of 20
|
||||
# The variable name starts with two underscores, which indicates private access
|
||||
self.__private_var = 20
|
||||
|
||||
# Create an object of MyClass class
|
||||
obj = MyClass()
|
||||
|
||||
# Access the protected variable using the object name and print its value
|
||||
# The protected variable can be accessed outside the class but
|
||||
# it is intended to be used within the class or its subclasses
|
||||
print(obj._protected_var) # output: 10
|
||||
|
||||
# Try to access the private variable using the object name and print its value
|
||||
# The private variable cannot be accessed outside the class, even by its subclasses
|
||||
# This will raise an AttributeError because the variable is not accessible outside the class
|
||||
print(obj.__private_var) # AttributeError: 'MyClass' object has no attribute '__private_var'
|
||||
|
||||
```
|
||||
|
||||
## Inheritance
|
||||
|
||||
Inheritance promotes code reuse and allows you to create a hierarchy of classes that share common attributes and methods. It helps in creating clean and organized code by keeping related functionality in one place and promoting the concept of modularity. The base class from which a new class is derived is also known as a parent class, and the new class is known as the child class or subclass.
|
||||
|
||||
In the code, we define a class named Animal which has a constructor method that initializes the class object with a name attribute and a method named speak. The speak method is defined in the Animal class but does not have a body.
|
||||
|
||||
We then define two subclasses named Dog and Cat which inherit from the Animal class. These subclasses override the speak method of the Animal class.
|
||||
|
||||
We create a Dog object with a name attribute "Rover" and a Cat object with a name attribute "Whiskers". We call the speak method of the Dog object using dog.speak(), and it prints "Woof!" because the speak method of the Dog class overrides the speak method of the Animal class. Similarly, we call the speak method of the Cat object using cat.speak(), and it prints "Meow!" because the speak method of the Cat class overrides the speak method of the Animal class.
|
||||
|
||||
``` python
|
||||
# Define a class named Animal
|
||||
class Animal:
|
||||
|
||||
# Constructor method that initializes the class object with a name attribute
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
# Method that is defined in the Animal class but does not have a body
|
||||
# This method will be overridden in the subclasses of Animal
|
||||
def speak(self):
|
||||
print("")
|
||||
|
||||
# Define a subclass named Dog that inherits from the Animal class
|
||||
class Dog(Animal):
|
||||
|
||||
# Override the speak method of the Animal class
|
||||
def speak(self):
|
||||
print("Woof!")
|
||||
|
||||
# Define a subclass named Cat that inherits from the Animal class
|
||||
class Cat(Animal):
|
||||
|
||||
# Override the speak method of the Animal class
|
||||
def speak(self):
|
||||
print("Meow!")
|
||||
|
||||
# Create a Dog object with a name attribute "Rover"
|
||||
dog = Dog("Rover")
|
||||
|
||||
# Create a Cat object with a name attribute "Whiskers"
|
||||
cat = Cat("Whiskers")
|
||||
|
||||
# Call the speak method of the Dog class and print the output
|
||||
# The speak method of the Dog class overrides the speak method of the Animal class
|
||||
# Therefore, when we call the speak method of the Dog object, it will print "Woof!"
|
||||
dog.speak() # output: Woof!
|
||||
|
||||
# Call the speak method of the Cat class and print the output
|
||||
# The speak method of the Cat class overrides the speak method of the Animal class
|
||||
# Therefore, when we call the speak method of the Cat object, it will print "Meow!"
|
||||
cat.speak() # output: Meow!
|
||||
|
||||
```
|
||||
|
||||
## Polymorphism
|
||||
|
||||
Polymorphism is an important concept in object-oriented programming that allows you to write code that can work with objects of different classes in a uniform way. In Python, polymorphism is achieved by using method overriding or method overloading.
|
||||
|
||||
Method overriding is when a subclass provides its own implementation of a method that is already defined in its parent class. This allows the subclass to modify the behavior of the method without changing its name or signature.
|
||||
|
||||
Method overloading is when multiple methods have the same name but different parameters. Python does not support method overloading directly, but it can be achieved using default arguments or variable-length arguments.
|
||||
|
||||
Polymorphism makes it easier to write flexible and reusable code. It allows you to write code that can work with different objects without needing to know their specific types.
|
||||
|
||||
```python
|
||||
#The Shape class is defined with an abstract area method, which is intended to be overridden by subclasses.
|
||||
class Shape:
|
||||
def area(self):
|
||||
pass
|
||||
|
||||
class Rectangle(Shape):
|
||||
# The Rectangle class is defined with an __init__ method that initializes
|
||||
# width and height instance variables.
|
||||
# It also defines an area method that calculates and returns
|
||||
# the area of a rectangle using the width and height instance variables.
|
||||
def __init__(self, width, height):
|
||||
self.width = width # Initialize width instance variable
|
||||
self.height = height # Initialize height instance variable
|
||||
|
||||
def area(self):
|
||||
return self.width * self.height # Return area of rectangle
|
||||
|
||||
|
||||
# The Circle class is defined with an __init__ method
|
||||
# that initializes a radius instance variable.
|
||||
# It also defines an area method that calculates and
|
||||
# returns the area of a circle using the radius instance variable.
|
||||
class Circle(Shape):
|
||||
def __init__(self, radius):
|
||||
self.radius = radius # Initialize radius instance variable
|
||||
|
||||
def area(self):
|
||||
return 3.14 * self.radius ** 2 # Return area of circle using pi * r^2
|
||||
|
||||
# The shapes list is created with one Rectangle object and one Circle object. The for
|
||||
# loop iterates over each object in the list and calls the area method of each object
|
||||
# The output will be the area of the rectangle (20) and the area of the circle (153.86).
|
||||
shapes = [Rectangle(4, 5), Circle(7)] # Create a list of Shape objects
|
||||
for shape in shapes:
|
||||
print(shape.area()) # Output the area of each Shape object
|
||||
|
||||
```
|
||||
|
||||
## Abstraction
|
||||
Abstraction is an important concept in object-oriented programming (OOP) because it allows you to focus on the essential features of an object or system while ignoring the details that aren't relevant to the current context. By reducing complexity and hiding unnecessary details, abstraction can make code more modular, easier to read, and easier to maintain.
|
||||
|
||||
In Python, abstraction can be achieved by using abstract classes or interfaces. An abstract class is a class that cannot be instantiated directly, but is meant to be subclassed by other classes. It often includes abstract methods that have no implementation, but provide a template for how the subclass should be implemented. This allows the programmer to define a common interface for a group of related classes, while still allowing each class to have its own specific behavior.
|
||||
|
||||
An interface, on the other hand, is a collection of method signatures that a class must implement in order to be considered "compatible" with the interface. Interfaces are often used to define a common set of methods that multiple classes can implement, allowing them to be used interchangeably in certain contexts.
|
||||
|
||||
Python does not have built-in support for abstract classes or interfaces, but they can be implemented using the abc (abstract base class) module. This module provides the ABC class and the abstractmethod decorator, which can be used to define abstract classes and methods.
|
||||
|
||||
Overall, abstraction is a powerful tool for managing complexity and improving code quality in object-oriented programming, and Python provides a range of options for achieving abstraction in your code.
|
||||
|
||||
```python
|
||||
# Import the abc module to define abstract classes and methods
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
# Define an abstract class called Shape that has an abstract method called area
|
||||
class Shape(ABC):
|
||||
@abstractmethod
|
||||
def area(self):
|
||||
pass
|
||||
|
||||
# Define a Rectangle class that inherits from Shape
|
||||
class Rectangle(Shape):
|
||||
def __init__(self, width, height):
|
||||
self.width = width
|
||||
self.height = height
|
||||
|
||||
# Implement the area method for Rectangles
|
||||
def area(self):
|
||||
return self.width * self.height
|
||||
|
||||
# Define a Circle class that also inherits from Shape
|
||||
class Circle(Shape):
|
||||
def __init__(self, radius):
|
||||
self.radius = radius
|
||||
|
||||
# Implement the area method for Circles
|
||||
def area(self):
|
||||
return 3.14 * self.radius ** 2
|
||||
|
||||
# Create a list of shapes that includes both Rectangles and Circles
|
||||
shapes = [Rectangle(4, 5), Circle(7)]
|
||||
|
||||
# Loop through each shape in the list and print its area
|
||||
for shape in shapes:
|
||||
print(shape.area())
|
||||
|
||||
```
|
||||
|
||||
These are some of the basic OOP principles in Python. This page is currently in progress and more
|
||||
detailed examples and explanations will be coming soon.
|
||||
@ -1,71 +0,0 @@
|
||||
---
|
||||
title: Reading and writing files - Python Cheatsheet
|
||||
description: To read/write to a file in Python, you will want to use the with statement, which will close the file for you after you are done, managing the available resources for you.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Reading and Writing Files
|
||||
</base-title>
|
||||
|
||||
## The file Reading/Writing process
|
||||
|
||||
To read/write to a file in Python, you will want to use the `with`
|
||||
statement, which will close the file for you after you are done, managing the available resources for you.
|
||||
|
||||
## Opening and reading files
|
||||
|
||||
The `open` function opens a file and return a corresponding file object.
|
||||
|
||||
```python
|
||||
>>> with open('C:\\Users\\your_home_folder\\hi.txt') as hello_file:
|
||||
... hello_content = hello_file.read()
|
||||
...
|
||||
>>> hello_content
|
||||
'Hello World!'
|
||||
```
|
||||
|
||||
Alternatively, you can use the _readlines()_ method to get a list of string values from the file, one string for each line of text:
|
||||
|
||||
```python
|
||||
>>> with open('sonnet29.txt') as sonnet_file:
|
||||
... sonnet_file.readlines()
|
||||
...
|
||||
# [When, in disgrace with fortune and men's eyes,\n',
|
||||
# ' I all alone beweep my outcast state,\n',
|
||||
# And trouble deaf heaven with my bootless cries,\n', And
|
||||
# look upon myself and curse my fate,']
|
||||
```
|
||||
|
||||
You can also iterate through the file line by line:
|
||||
|
||||
```python
|
||||
>>> with open('sonnet29.txt') as sonnet_file:
|
||||
... for line in sonnet_file:
|
||||
... print(line, end='')
|
||||
...
|
||||
# When, in disgrace with fortune and men's eyes,
|
||||
# I all alone beweep my outcast state,
|
||||
# And trouble deaf heaven with my bootless cries,
|
||||
# And look upon myself and curse my fate,
|
||||
```
|
||||
|
||||
## Writing to files
|
||||
|
||||
```python
|
||||
>>> with open('bacon.txt', 'w') as bacon_file:
|
||||
... bacon_file.write('Hello world!\n')
|
||||
...
|
||||
# 13
|
||||
|
||||
>>> with open('bacon.txt', 'a') as bacon_file:
|
||||
... bacon_file.write('Bacon is not a vegetable.')
|
||||
...
|
||||
# 25
|
||||
|
||||
>>> with open('bacon.txt') as bacon_file:
|
||||
... content = bacon_file.read()
|
||||
...
|
||||
>>> print(content)
|
||||
# Hello world!
|
||||
# Bacon is not a vegetable.
|
||||
```
|
||||
@ -1,393 +0,0 @@
|
||||
---
|
||||
title: Python Regular Expressions - Python Cheatsheet
|
||||
description: A regular expression (shortened as regex) is a sequence of characters that specifies a search pattern in text and used by string-searching algorithms.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Regular Expressions
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="_blank" href="https://en.wikipedia.org/wiki/Regular_expression">Regular expressions</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
A regular expression (shortened as regex [...]) is a sequence of characters that specifies a search pattern in text. [...] used by string-searching algorithms for "find" or "find and replace" operations on strings, or for input validation.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
1. Import the regex module with `import re`.
|
||||
2. Create a Regex object with the `re.compile()` function. (Remember to use a raw string.)
|
||||
3. Pass the string you want to search into the Regex object’s `search()` method. This returns a `Match` object.
|
||||
4. Call the Match object’s `group()` method to return a string of the actual matched text.
|
||||
|
||||
All the regex functions in Python are in the re module:
|
||||
|
||||
```python
|
||||
>>> import re
|
||||
```
|
||||
|
||||
## Regex symbols
|
||||
|
||||
| Symbol | Matches |
|
||||
| ------------------------ | ------------------------------------------------------ |
|
||||
| `?` | zero or one of the preceding group. |
|
||||
| `*` | zero or more of the preceding group. |
|
||||
| `+` | one or more of the preceding group. |
|
||||
| `{n}` | exactly n of the preceding group. |
|
||||
| `{n,}` | n or more of the preceding group. |
|
||||
| `{,m}` | 0 to m of the preceding group. |
|
||||
| `{n,m}` | at least n and at most m of the preceding p. |
|
||||
| `{n,m}?` or `*?` or `+?` | performs a non-greedy match of the preceding p. |
|
||||
| `^spam` | means the string must begin with spam. |
|
||||
| `spam$` | means the string must end with spam. |
|
||||
| `.` | any character, except newline characters. |
|
||||
| `\d`, `\w`, and `\s` | a digit, word, or space character, respectively. |
|
||||
| `\D`, `\W`, and `\S` | anything except a digit, word, or space, respectively. |
|
||||
| `[abc]` | any character between the brackets (such as a, b, ). |
|
||||
| `[^abc]` | any character that isn’t between the brackets. |
|
||||
|
||||
## Matching regex objects
|
||||
|
||||
```python
|
||||
>>> phone_num_regex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d')
|
||||
|
||||
>>> mo = phone_num_regex.search('My number is 415-555-4242.')
|
||||
|
||||
>>> print(f'Phone number found: {mo.group()}')
|
||||
# Phone number found: 415-555-4242
|
||||
```
|
||||
|
||||
## Grouping with parentheses
|
||||
|
||||
```python
|
||||
>>> phone_num_regex = re.compile(r'(\d\d\d)-(\d\d\d-\d\d\d\d)')
|
||||
>>> mo = phone_num_regex.search('My number is 415-555-4242.')
|
||||
|
||||
>>> mo.group(1)
|
||||
# '415'
|
||||
|
||||
>>> mo.group(2)
|
||||
# '555-4242'
|
||||
|
||||
>>> mo.group(0)
|
||||
# '415-555-4242'
|
||||
|
||||
>>> mo.group()
|
||||
# '415-555-4242'
|
||||
```
|
||||
|
||||
To retrieve all the groups at once use the `groups()` method:
|
||||
|
||||
```python
|
||||
>>> mo.groups()
|
||||
('415', '555-4242')
|
||||
|
||||
>>> area_code, main_number = mo.groups()
|
||||
|
||||
>>> print(area_code)
|
||||
415
|
||||
|
||||
>>> print(main_number)
|
||||
555-4242
|
||||
```
|
||||
|
||||
## Multiple groups with Pipe
|
||||
|
||||
You can use the `|` character anywhere you want to match one of many expressions.
|
||||
|
||||
```python
|
||||
>>> hero_regex = re.compile (r'Batman|Tina Fey')
|
||||
|
||||
>>> mo1 = hero_regex.search('Batman and Tina Fey.')
|
||||
>>> mo1.group()
|
||||
# 'Batman'
|
||||
|
||||
>>> mo2 = hero_regex.search('Tina Fey and Batman.')
|
||||
>>> mo2.group()
|
||||
# 'Tina Fey'
|
||||
```
|
||||
|
||||
You can also use the pipe to match one of several patterns as part of your regex:
|
||||
|
||||
```python
|
||||
>>> bat_regex = re.compile(r'Bat(man|mobile|copter|bat)')
|
||||
>>> mo = bat_regex.search('Batmobile lost a wheel')
|
||||
|
||||
>>> mo.group()
|
||||
# 'Batmobile'
|
||||
|
||||
>>> mo.group(1)
|
||||
# 'mobile'
|
||||
```
|
||||
|
||||
## Optional matching with the Question Mark
|
||||
|
||||
The `?` character flags the group that precedes it as an optional part of the pattern.
|
||||
|
||||
```python
|
||||
>>> bat_regex = re.compile(r'Bat(wo)?man')
|
||||
|
||||
>>> mo1 = bat_regex.search('The Adventures of Batman')
|
||||
>>> mo1.group()
|
||||
# 'Batman'
|
||||
|
||||
>>> mo2 = bat_regex.search('The Adventures of Batwoman')
|
||||
>>> mo2.group()
|
||||
# 'Batwoman'
|
||||
```
|
||||
|
||||
## Matching zero or more with the Star
|
||||
|
||||
The `*` (star or asterisk) means “match zero or more”. The group that precedes the star can occur any number of times in the text.
|
||||
|
||||
```python
|
||||
>>> bat_regex = re.compile(r'Bat(wo)*man')
|
||||
>>> mo1 = bat_regex.search('The Adventures of Batman')
|
||||
>>> mo1.group()
|
||||
'Batman'
|
||||
|
||||
>>> mo2 = bat_regex.search('The Adventures of Batwoman')
|
||||
>>> mo2.group()
|
||||
'Batwoman'
|
||||
|
||||
>>> mo3 = bat_regex.search('The Adventures of Batwowowowoman')
|
||||
>>> mo3.group()
|
||||
'Batwowowowoman'
|
||||
```
|
||||
|
||||
## Matching one or more with the Plus
|
||||
|
||||
The `+` (or plus) _means match one or more_. The group preceding a plus must appear at least once:
|
||||
|
||||
```python
|
||||
>>> bat_regex = re.compile(r'Bat(wo)+man')
|
||||
|
||||
>>> mo1 = bat_regex.search('The Adventures of Batwoman')
|
||||
>>> mo1.group()
|
||||
# 'Batwoman'
|
||||
|
||||
>>> mo2 = bat_regex.search('The Adventures of Batwowowowoman')
|
||||
>>> mo2.group()
|
||||
# 'Batwowowowoman'
|
||||
|
||||
>>> mo3 = bat_regex.search('The Adventures of Batman')
|
||||
>>> mo3 is None
|
||||
# True
|
||||
```
|
||||
|
||||
## Matching specific repetitions with Curly Brackets
|
||||
|
||||
If you have a group that you want to repeat a specific number of times, follow the group in your regex with a number in curly brackets:
|
||||
|
||||
```python
|
||||
>>> ha_regex = re.compile(r'(Ha){3}')
|
||||
|
||||
>>> mo1 = ha_regex.search('HaHaHa')
|
||||
>>> mo1.group()
|
||||
# 'HaHaHa'
|
||||
|
||||
>>> mo2 = ha_regex.search('Ha')
|
||||
>>> mo2 is None
|
||||
# True
|
||||
```
|
||||
|
||||
Instead of one number, you can specify a range with minimum and a maximum in between the curly brackets. For example, the regex (Ha){3,5} will match 'HaHaHa', 'HaHaHaHa', and 'HaHaHaHaHa'.
|
||||
|
||||
```python
|
||||
>>> ha_regex = re.compile(r'(Ha){2,3}')
|
||||
>>> mo1 = ha_regex.search('HaHaHaHa')
|
||||
>>> mo1.group()
|
||||
# 'HaHaHa'
|
||||
```
|
||||
|
||||
## Greedy and non-greedy matching
|
||||
|
||||
Python’s regular expressions are greedy by default: in ambiguous situations they will match the longest string possible. The non-greedy version of the curly brackets, which matches the shortest string possible, has the closing curly bracket followed by a question mark.
|
||||
|
||||
```python
|
||||
>>> greedy_ha_regex = re.compile(r'(Ha){3,5}')
|
||||
|
||||
>>> mo1 = greedy_ha_regex.search('HaHaHaHaHa')
|
||||
>>> mo1.group()
|
||||
# 'HaHaHaHaHa'
|
||||
|
||||
>>> non_greedy_ha_regex = re.compile(r'(Ha){3,5}?')
|
||||
>>> mo2 = non_greedy_ha_regex.search('HaHaHaHaHa')
|
||||
>>> mo2.group()
|
||||
# 'HaHaHa'
|
||||
```
|
||||
|
||||
## The findall() method
|
||||
|
||||
The `findall()` method will return the strings of every match in the searched string.
|
||||
|
||||
```python
|
||||
>>> phone_num_regex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d') # has no groups
|
||||
|
||||
>>> phone_num_regex.findall('Cell: 415-555-9999 Work: 212-555-0000')
|
||||
# ['415-555-9999', '212-555-0000']
|
||||
```
|
||||
|
||||
## Making your own character classes
|
||||
|
||||
You can define your own character class using square brackets. For example, the character class _[aeiouAEIOU]_ will match any vowel, both lowercase and uppercase.
|
||||
|
||||
```python
|
||||
>>> vowel_regex = re.compile(r'[aeiouAEIOU]')
|
||||
>>> vowel_regex.findall('Robocop eats baby food. BABY FOOD.')
|
||||
# ['o', 'o', 'o', 'e', 'a', 'a', 'o', 'o', 'A', 'O', 'O']
|
||||
```
|
||||
|
||||
You can also include ranges of letters or numbers by using a hyphen. For example, the character class _[a-zA-Z0-9]_ will match all lowercase letters, uppercase letters, and numbers.
|
||||
|
||||
By placing a caret character (`^`) just after the character class’s opening bracket, you can make a negative character class that will match all the characters that are not in the character class:
|
||||
|
||||
```python
|
||||
>>> consonant_regex = re.compile(r'[^aeiouAEIOU]')
|
||||
>>> consonant_regex.findall('Robocop eats baby food. BABY FOOD.')
|
||||
# ['R', 'b', 'c', 'p', ' ', 't', 's', ' ', 'b', 'b', 'y', ' ', 'f', 'd', '.', '
|
||||
# ', 'B', 'B', 'Y', ' ', 'F', 'D', '.']
|
||||
```
|
||||
|
||||
## The Caret and Dollar sign characters
|
||||
|
||||
- You can also use the caret symbol `^` at the start of a regex to indicate that a match must occur at the beginning of the searched text.
|
||||
|
||||
- Likewise, you can put a dollar sign `$` at the end of the regex to indicate the string must end with this regex pattern.
|
||||
|
||||
- And you can use the `^` and `$` together to indicate that the entire string must match the regex.
|
||||
|
||||
The `r'^Hello`' regular expression string matches strings that begin with 'Hello':
|
||||
|
||||
```python
|
||||
>>> begins_with_hello = re.compile(r'^Hello')
|
||||
>>> begins_with_hello.search('Hello world!')
|
||||
# <_sre.SRE_Match object; span=(0, 5), match='Hello'>
|
||||
|
||||
>>> begins_with_hello.search('He said hello.') is None
|
||||
# True
|
||||
```
|
||||
|
||||
The `r'\d\$'` regular expression string matches strings that end with a numeric character from 0 to 9:
|
||||
|
||||
```python
|
||||
>>> whole_string_is_num = re.compile(r'^\d+$')
|
||||
|
||||
>>> whole_string_is_num.search('1234567890')
|
||||
# <_sre.SRE_Match object; span=(0, 10), match='1234567890'>
|
||||
|
||||
>>> whole_string_is_num.search('12345xyz67890') is None
|
||||
# True
|
||||
|
||||
>>> whole_string_is_num.search('12 34567890') is None
|
||||
# True
|
||||
```
|
||||
|
||||
## The Wildcard character
|
||||
|
||||
The `.` (or dot) character in a regular expression will match any character except for a newline:
|
||||
|
||||
```python
|
||||
>>> at_regex = re.compile(r'.at')
|
||||
|
||||
>>> at_regex.findall('The cat in the hat sat on the flat mat.')
|
||||
['cat', 'hat', 'sat', 'lat', 'mat']
|
||||
```
|
||||
|
||||
## Matching everything with Dot-Star
|
||||
|
||||
```python
|
||||
>>> name_regex = re.compile(r'First Name: (.*) Last Name: (.*)')
|
||||
|
||||
>>> mo = name_regex.search('First Name: Al Last Name: Sweigart')
|
||||
>>> mo.group(1)
|
||||
# 'Al'
|
||||
|
||||
>>> mo.group(2)
|
||||
'Sweigart'
|
||||
```
|
||||
|
||||
The `.*` uses greedy mode: It will always try to match as much text as possible. To match any and all text in a non-greedy fashion, use the dot, star, and question mark (`.*?`). The question mark tells Python to match in a non-greedy way:
|
||||
|
||||
```python
|
||||
>>> non_greedy_regex = re.compile(r'<.*?>')
|
||||
>>> mo = non_greedy_regex.search('<To serve man> for dinner.>')
|
||||
>>> mo.group()
|
||||
# '<To serve man>'
|
||||
|
||||
>>> greedy_regex = re.compile(r'<.*>')
|
||||
>>> mo = greedy_regex.search('<To serve man> for dinner.>')
|
||||
>>> mo.group()
|
||||
# '<To serve man> for dinner.>'
|
||||
```
|
||||
|
||||
## Matching newlines with the Dot character
|
||||
|
||||
The dot-star will match everything except a newline. By passing `re.DOTALL` as the second argument to `re.compile()`, you can make the dot character match all characters, including the newline character:
|
||||
|
||||
```python
|
||||
>>> no_newline_regex = re.compile('.*')
|
||||
>>> no_newline_regex.search('Serve the public trust.\nProtect the innocent.\nUphold the law.').group()
|
||||
# 'Serve the public trust.'
|
||||
|
||||
>>> newline_regex = re.compile('.*', re.DOTALL)
|
||||
>>> newline_regex.search('Serve the public trust.\nProtect the innocent.\nUphold the law.').group()
|
||||
# 'Serve the public trust.\nProtect the innocent.\nUphold the law.'
|
||||
```
|
||||
|
||||
## Case-Insensitive matching
|
||||
|
||||
To make your regex case-insensitive, you can pass `re.IGNORECASE` or `re.I` as a second argument to `re.compile()`:
|
||||
|
||||
```python
|
||||
>>> robocop = re.compile(r'robocop', re.I)
|
||||
|
||||
>>> robocop.search('Robocop is part man, part machine, all cop.').group()
|
||||
# 'Robocop'
|
||||
|
||||
>>> robocop.search('ROBOCOP protects the innocent.').group()
|
||||
# 'ROBOCOP'
|
||||
|
||||
>>> robocop.search('Al, why does your programming book talk about robocop so much?').group()
|
||||
# 'robocop'
|
||||
```
|
||||
|
||||
## Substituting strings with the sub() method
|
||||
|
||||
The `sub()` method for Regex objects is passed two arguments:
|
||||
|
||||
1. The first argument is a string to replace any matches.
|
||||
1. The second is the string for the regular expression.
|
||||
|
||||
The `sub()` method returns a string with the substitutions applied:
|
||||
|
||||
```python
|
||||
>>> names_regex = re.compile(r'Agent \w+')
|
||||
|
||||
>>> names_regex.sub('CENSORED', 'Agent Alice gave the secret documents to Agent Bob.')
|
||||
# 'CENSORED gave the secret documents to CENSORED.'
|
||||
```
|
||||
|
||||
## Managing complex Regexes
|
||||
|
||||
To tell the `re.compile()` function to ignore whitespace and comments inside the regular expression string, “verbose mode” can be enabled by passing the variable `re.VERBOSE` as the second argument to `re.compile()`.
|
||||
|
||||
Now instead of a hard-to-read regular expression like this:
|
||||
|
||||
```python
|
||||
phone_regex = re.compile(r'((\d{3}|\(\d{3}\))?(\s|-|\.)?\d{3}(\s|-|\.)\d{4}(\s*(ext|x|ext.)\s*\d{2,5})?)')
|
||||
```
|
||||
|
||||
you can spread the regular expression over multiple lines with comments like this:
|
||||
|
||||
```python
|
||||
phone_regex = re.compile(r'''(
|
||||
(\d{3}|\(\d{3}\))? # area code
|
||||
(\s|-|\.)? # separator
|
||||
\d{3} # first 3 digits
|
||||
(\s|-|\.) # separator
|
||||
\d{4} # last 4 digits
|
||||
(\s*(ext|x|ext.)\s*\d{2,5})? # extension
|
||||
)''', re.VERBOSE)
|
||||
```
|
||||
@ -1,158 +0,0 @@
|
||||
---
|
||||
title: Python Sets - Python Cheatsheet
|
||||
description: Python comes equipped with several built-in data types to help us organize our data. These structures include lists, dictionaries, tuples and sets.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python Sets
|
||||
</base-title>
|
||||
|
||||
Python comes equipped with several built-in data types to help us organize our data. These structures include lists, dictionaries, tuples and **sets**.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the Python 3 <a target="_blank" href="https://docs.python.org/3/tutorial/datastructures.html#sets">documentation</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
A set is an unordered collection with no duplicate elements. Basic uses include membership testing and eliminating duplicate entries.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Read <router-link to="/blog/python-sets-what-why-how">Python Sets: What, Why and How</router-link> for a more in-deep reference.
|
||||
|
||||
## Initializing a set
|
||||
|
||||
There are two ways to create sets: using curly braces `{}` and the built-in function `set()`
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
Empty Sets
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
When creating set, be sure to not use empty curly braces <code>{}</code> or you will get an empty dictionary instead.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s = set([1, 2, 3])
|
||||
|
||||
>>> s = {} # this will create a dictionary instead of a set
|
||||
>>> type(s)
|
||||
# <class 'dict'>
|
||||
```
|
||||
|
||||
## Unordered collections of unique elements
|
||||
|
||||
A set automatically removes all the duplicate values.
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3, 2, 3, 4}
|
||||
>>> s
|
||||
# {1, 2, 3, 4}
|
||||
```
|
||||
|
||||
And as an unordered data type, they can't be indexed.
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s[0]
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# TypeError: 'set' object does not support indexing
|
||||
```
|
||||
|
||||
## set add and update
|
||||
|
||||
Using the `add()` method we can add a single element to the set.
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s.add(4)
|
||||
>>> s
|
||||
# {1, 2, 3, 4}
|
||||
```
|
||||
|
||||
And with `update()`, multiple ones:
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s.update([2, 3, 4, 5, 6])
|
||||
>>> s
|
||||
# {1, 2, 3, 4, 5, 6}
|
||||
```
|
||||
|
||||
## set remove and discard
|
||||
|
||||
Both methods will remove an element from the set, but `remove()` will raise a `key error` if the value doesn't exist.
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s.remove(3)
|
||||
>>> s
|
||||
# {1, 2}
|
||||
|
||||
>>> s.remove(3)
|
||||
# Traceback (most recent call last):
|
||||
# File "<stdin>", line 1, in <module>
|
||||
# KeyError: 3
|
||||
```
|
||||
|
||||
`discard()` won't raise any errors.
|
||||
|
||||
```python
|
||||
>>> s = {1, 2, 3}
|
||||
>>> s.discard(3)
|
||||
>>> s
|
||||
# {1, 2}
|
||||
>>> s.discard(3)
|
||||
```
|
||||
|
||||
## set union
|
||||
|
||||
`union()` or `|` will create a new set with all the elements from the sets provided.
|
||||
|
||||
```python
|
||||
>>> s1 = {1, 2, 3}
|
||||
>>> s2 = {3, 4, 5}
|
||||
>>> s1.union(s2) # or 's1 | s2'
|
||||
# {1, 2, 3, 4, 5}
|
||||
```
|
||||
|
||||
## set intersection
|
||||
|
||||
`intersection()` or `&` will return a set with only the elements that are common to all of them.
|
||||
|
||||
```python
|
||||
>>> s1 = {1, 2, 3}
|
||||
>>> s2 = {2, 3, 4}
|
||||
>>> s3 = {3, 4, 5}
|
||||
>>> s1.intersection(s2, s3) # or 's1 & s2 & s3'
|
||||
# {3}
|
||||
```
|
||||
|
||||
## set difference
|
||||
|
||||
`difference()` or `-` will return only the elements that are unique to the first set (invoked set).
|
||||
|
||||
```python
|
||||
>>> s1 = {1, 2, 3}
|
||||
>>> s2 = {2, 3, 4}
|
||||
|
||||
>>> s1.difference(s2) # or 's1 - s2'
|
||||
# {1}
|
||||
|
||||
>>> s2.difference(s1) # or 's2 - s1'
|
||||
# {4}
|
||||
```
|
||||
|
||||
## set symmetric_difference
|
||||
|
||||
`symmetric_difference()` or `^` will return all the elements that are not common between them.
|
||||
|
||||
```python
|
||||
>>> s1 = {1, 2, 3}
|
||||
>>> s2 = {2, 3, 4}
|
||||
>>> s1.symmetric_difference(s2) # or 's1 ^ s2'
|
||||
# {1, 4}
|
||||
```
|
||||
@ -1,54 +0,0 @@
|
||||
---
|
||||
title: Python Setup.py - Python Cheatsheet
|
||||
description: The setup script is the centre of all activity in building, distributing, and installing modules using the Distutils. The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python setup.py
|
||||
</base-title>
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
A 'controversial' opinion
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
Using `setup.py` to pack and distribute your python packages can be quite challenging every so often. Tools like <a target="_blank" href="https://python-poetry.org/">Poetry</a> make not only the packaging a <b>lot easier</b>, but also help you to manage your dependencies in a very convenient way.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
If you want more information about Poetry you can read the following articles:
|
||||
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-1">Python projects with Poetry and VSCode. Part 1</router-link>
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-2">Python projects with Poetry and VSCode. Part 2</router-link>
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-3">Python projects with Poetry and VSCode. Part 3</router-link>
|
||||
|
||||
## Introduction
|
||||
|
||||
The setup script is the center of all activity in building, distributing, and installing modules using the Distutils. The main purpose of the setup script is to describe your module distribution to the Distutils, so that the various commands that operate on your modules do the right thing.
|
||||
|
||||
The `setup.py` file is at the heart of a Python project. It describes all the metadata about your project. There are quite a few fields you can add to a project to give it a rich set of metadata describing the project. However, there are only three required fields: name, version, and packages. The name field must be unique if you wish to publish your package on the Python Package Index (PyPI). The version field keeps track of different releases of the project. The package's field describes where you’ve put the Python source code within your project.
|
||||
|
||||
This allows you to easily install Python packages. Often it's enough to write:
|
||||
|
||||
```bash
|
||||
python setup.py install
|
||||
```
|
||||
|
||||
and module will install itself.
|
||||
|
||||
## Example
|
||||
|
||||
Our initial setup.py will also include information about the license and will re-use the README.txt file for the long_description field. This will look like:
|
||||
|
||||
```python
|
||||
from distutils.core import setup
|
||||
setup(
|
||||
name='pythonCheatsheet',
|
||||
version='0.1',
|
||||
packages=['pipenv',],
|
||||
license='MIT',
|
||||
long_description=open('README.txt').read(),
|
||||
)
|
||||
```
|
||||
|
||||
Find more information visit the [official documentation](http://docs.python.org/3.11/install/index.html).
|
||||
@ -1,177 +0,0 @@
|
||||
---
|
||||
title: Python String Formatting - Python Cheatsheet
|
||||
description: If your are using Python 3.6+, string f-strings are the recommended way to format strings.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Python String Formatting
|
||||
</base-title>
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the <a href="https://docs.python.org/3/library/stdtypes.html?highlight=sprintf#printf-style-string-formatting">Python 3 documentation</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
The formatting operations described here (<b>% operator</b>) exhibit a variety of quirks that lead to a number of common errors [...]. Using the newer <a href="#formatted-string-literals-or-f-strings">formatted string literals</a> [...] helps avoid these errors. These alternatives also provide more powerful, flexible and extensible approaches to formatting text.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
## % operator
|
||||
|
||||
<base-warning>
|
||||
<base-warning-title>
|
||||
Prefer String Literals
|
||||
</base-warning-title>
|
||||
<base-warning-content>
|
||||
For new code, using <a href="#strformat">str.format</a>, or <a href="#formatted-string-literals-or-f-strings">formatted string literals</a> (Python 3.6+) over the <code>%</code> operator is strongly recommended.
|
||||
</base-warning-content>
|
||||
</base-warning>
|
||||
|
||||
```python
|
||||
>>> name = 'Pete'
|
||||
>>> 'Hello %s' % name
|
||||
# "Hello Pete"
|
||||
```
|
||||
|
||||
We can use the `%d` format specifier to convert an int value to a string:
|
||||
|
||||
```python
|
||||
>>> num = 5
|
||||
>>> 'I have %d apples' % num
|
||||
# "I have 5 apples"
|
||||
```
|
||||
|
||||
## str.format
|
||||
|
||||
Python 3 introduced a new way to do string formatting that was later back-ported to Python 2.7. This makes the syntax for string formatting more regular.
|
||||
|
||||
```python
|
||||
>>> name = 'John'
|
||||
>>> age = 20
|
||||
|
||||
>>> "Hello I'm {}, my age is {}".format(name, age)
|
||||
# "Hello I'm John, my age is 20"
|
||||
|
||||
>>> "Hello I'm {0}, my age is {1}".format(name, age)
|
||||
# "Hello I'm John, my age is 20"
|
||||
```
|
||||
|
||||
## Formatted String Literals or f-Strings
|
||||
|
||||
If your are using Python 3.6+, string `f-Strings` are the recommended way to format strings.
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From the <a href="https://docs.python.org/3/reference/lexical_analysis.html#f-strings">Python 3 documentation</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
A formatted string literal or f-string is a string literal that is prefixed with `f` or `F`. These strings may contain replacement fields, which are expressions delimited by curly braces {}. While other string literals always have a constant value, formatted strings are really expressions evaluated at run time.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
```python
|
||||
>>> name = 'Elizabeth'
|
||||
>>> f'Hello {name}!'
|
||||
# 'Hello Elizabeth!'
|
||||
```
|
||||
|
||||
It is even possible to do inline arithmetic with it:
|
||||
|
||||
```python
|
||||
>>> a = 5
|
||||
>>> b = 10
|
||||
>>> f'Five plus ten is {a + b} and not {2 * (a + b)}.'
|
||||
# 'Five plus ten is 15 and not 30.'
|
||||
```
|
||||
|
||||
### Multiline f-Strings
|
||||
|
||||
```python
|
||||
>>> name = 'Robert'
|
||||
>>> messages = 12
|
||||
>>> (
|
||||
... f'Hi, {name}. '
|
||||
... f'You have {messages} unread messages'
|
||||
... )
|
||||
# 'Hi, Robert. You have 12 unread messages'
|
||||
```
|
||||
|
||||
### The `=` specifier
|
||||
|
||||
This will print the expression and its value:
|
||||
|
||||
```python
|
||||
>>> from datetime import datetime
|
||||
>>> now = datetime.now().strftime("%b/%d/%Y - %H:%M:%S")
|
||||
>>> f'date and time: {now=}'
|
||||
# "date and time: now='Nov/14/2022 - 20:50:01'"
|
||||
```
|
||||
|
||||
### Adding spaces or characters
|
||||
|
||||
```python
|
||||
>>> f"{name.upper() = :-^20}"
|
||||
# 'name.upper() = -------ROBERT-------'
|
||||
>>>
|
||||
>>> f"{name.upper() = :^20}"
|
||||
# 'name.upper() = ROBERT '
|
||||
>>>
|
||||
>>> f"{name.upper() = :20}"
|
||||
# 'name.upper() = ROBERT '
|
||||
```
|
||||
|
||||
## Formatting Digits
|
||||
|
||||
Adding thousands separator
|
||||
|
||||
```python
|
||||
>>> a = 10000000
|
||||
>>> f"{a:,}"
|
||||
# '10,000,000'
|
||||
```
|
||||
|
||||
Rounding
|
||||
|
||||
```python
|
||||
>>> a = 3.1415926
|
||||
>>> f"{a:.2f}"
|
||||
# '3.14'
|
||||
```
|
||||
|
||||
Showing as Percentage
|
||||
|
||||
```python
|
||||
>>> a = 0.816562
|
||||
>>> f"{a:.2%}"
|
||||
# '81.66%'
|
||||
```
|
||||
|
||||
### Number formatting table
|
||||
|
||||
| Number | Format | Output | description |
|
||||
| ---------- | ------- | --------- | --------------------------------------------- |
|
||||
| 3.1415926 | {:.2f} | 3.14 | Format float 2 decimal places |
|
||||
| 3.1415926 | {:+.2f} | +3.14 | Format float 2 decimal places with sign |
|
||||
| -1 | {:+.2f} | -1.00 | Format float 2 decimal places with sign |
|
||||
| 2.71828 | {:.0f} | 3 | Format float with no decimal places |
|
||||
| 4 | {:0>2d} | 04 | Pad number with zeros (left padding, width 2) |
|
||||
| 4 | {:x<4d} | 4xxx | Pad number with x’s (right padding, width 4) |
|
||||
| 10 | {:x<4d} | 10xx | Pad number with x’s (right padding, width 4) |
|
||||
| 1000000 | {:,} | 1,000,000 | Number format with comma separator |
|
||||
| 0.35 | {:.2%} | 35.00% | Format percentage |
|
||||
| 1000000000 | {:.2e} | 1.00e+09 | Exponent notation |
|
||||
| 11 | {:11d} | 11 | Right-aligned (default, width 10) |
|
||||
| 11 | {:<11d} | 11 | Left-aligned (width 10) |
|
||||
| 11 | {:^11d} | 11 | Center aligned (width 10) |
|
||||
|
||||
## Template Strings
|
||||
|
||||
A simpler and less powerful mechanism, but it is recommended when handling strings generated by users. Due to their reduced complexity, template strings are a safer choice.
|
||||
|
||||
```python
|
||||
>>> from string import Template
|
||||
>>> name = 'Elizabeth'
|
||||
>>> t = Template('Hey $name!')
|
||||
>>> t.substitute(name=name)
|
||||
# 'Hey Elizabeth!'
|
||||
```
|
||||
@ -1,180 +0,0 @@
|
||||
---
|
||||
title: Python Virtual environments - Python Cheatsheet
|
||||
description: The use of a Virtual Environment is to test python code in encapsulated environments and to also avoid filling the base Python installation with libraries we might use for only one project.
|
||||
---
|
||||
|
||||
<base-title :title="frontmatter.title" :description="frontmatter.description">
|
||||
Virtual Environment
|
||||
</base-title>
|
||||
|
||||
The use of a Virtual Environment is to test python code in encapsulated environments, and to also avoid filling the base Python installation with libraries we might use for only one project.
|
||||
|
||||
## virtualenv
|
||||
|
||||
1. Install virtualenv
|
||||
|
||||
pip install virtualenv
|
||||
|
||||
1. Install virtualenvwrapper-win (Windows)
|
||||
|
||||
pip install virtualenvwrapper-win
|
||||
|
||||
Usage:
|
||||
|
||||
1. Make a Virtual Environment named `HelloWorld`
|
||||
|
||||
mkvirtualenv HelloWorld
|
||||
|
||||
Anything we install now will be specific to this project. And available to the projects we connect to this environment.
|
||||
|
||||
1. Set Project Directory
|
||||
|
||||
To bind our virtualenv with our current working directory we simply enter:
|
||||
|
||||
setprojectdir .
|
||||
|
||||
1. Deactivate
|
||||
|
||||
To move onto something else in the command line type `deactivate` to deactivate your environment.
|
||||
|
||||
deactivate
|
||||
|
||||
Notice how the parenthesis disappear.
|
||||
|
||||
1. Workon
|
||||
|
||||
Open up the command prompt and type `workon HelloWorld` to activate the environment and move into your root project folder
|
||||
|
||||
workon HelloWorld
|
||||
|
||||
## Poetry
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From <a href="https://python-poetry.org/">Poetry website</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
1. Install Poetry
|
||||
|
||||
pip install --user poetry
|
||||
|
||||
2. Create a new project
|
||||
|
||||
poetry new my-project
|
||||
|
||||
This will create a my-project directory:
|
||||
|
||||
my-project
|
||||
├── pyproject.toml
|
||||
├── README.rst
|
||||
├── poetry_demo
|
||||
│ └── __init__.py
|
||||
└── tests
|
||||
├── __init__.py
|
||||
└── test_poetry_demo.py
|
||||
|
||||
The pyproject.toml file will orchestrate your project and its dependencies:
|
||||
|
||||
[tool.poetry]
|
||||
name = "my-project"
|
||||
version = "0.1.0"
|
||||
description = ""
|
||||
authors = ["your name <your@mail.com>"]
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
python = "*"
|
||||
|
||||
[tool.poetry.dev-dependencies]
|
||||
pytest = "^3.4"
|
||||
|
||||
3. Packages
|
||||
|
||||
To add dependencies to your project, you can specify them in the tool.poetry.dependencies section:
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
pendulum = "^1.4"
|
||||
|
||||
Also, instead of modifying the pyproject.toml file by hand, you can use the add command and it will automatically find a suitable version constraint.
|
||||
|
||||
$ poetry add pendulum
|
||||
|
||||
To install the dependencies listed in the pyproject.toml:
|
||||
|
||||
poetry install
|
||||
|
||||
To remove dependencies:
|
||||
|
||||
poetry remove pendulum
|
||||
|
||||
For more information, check the [documentation](https://poetry.eustace.io/docs/) or read here:
|
||||
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-1">Python projects with Poetry and VSCode. Part 1</router-link>
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-2">Python projects with Poetry and VSCode. Part 2</router-link>
|
||||
- <router-link to="/blog/python-projects-with-poetry-and-vscode-part-3">Python projects with Poetry and VSCode. Part 3</router-link>
|
||||
|
||||
## Pipenv
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
From <a target="_blank" href="https://pipenv.pypa.io/en/latest/">Pipenv website</a>
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Pipenv is a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world. Windows is a first-class citizen, in our world.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
1. Install pipenv
|
||||
|
||||
pip install pipenv
|
||||
|
||||
2. Enter your Project directory and install the Packages for your project
|
||||
|
||||
cd my_project
|
||||
pipenv install <package>
|
||||
|
||||
Pipenv will install your package and create a Pipfile for you in your project’s directory. The Pipfile is used to track which dependencies your project needs in case you need to re-install them.
|
||||
|
||||
3. Uninstall Packages
|
||||
|
||||
pipenv uninstall <package>
|
||||
|
||||
4. Activate the Virtual Environment associated with your Python project
|
||||
|
||||
pipenv shell
|
||||
|
||||
5. Exit the Virtual Environment
|
||||
|
||||
exit
|
||||
|
||||
Find more information and a video in [docs.pipenv.org](https://docs.pipenv.org/).
|
||||
|
||||
## Anaconda
|
||||
|
||||
<base-disclaimer>
|
||||
<base-disclaimer-title>
|
||||
<a target="k" href="https://anaconda.com/">Anaconda</a> is another popular tool to manage python packages.
|
||||
</base-disclaimer-title>
|
||||
<base-disclaimer-content>
|
||||
Where packages, notebooks, projects and environments are shared. Your place for free public conda package hosting.
|
||||
</base-disclaimer-content>
|
||||
</base-disclaimer>
|
||||
|
||||
Usage:
|
||||
|
||||
1. Make a Virtual Environment
|
||||
|
||||
conda create -n HelloWorld
|
||||
|
||||
2. To use the Virtual Environment, activate it by:
|
||||
|
||||
conda activate HelloWorld
|
||||
|
||||
Anything installed now will be specific to the project HelloWorld
|
||||
|
||||
3. Exit the Virtual Environment
|
||||
|
||||
conda deactivate
|
||||
29
README.md
Normal file
29
README.md
Normal file
@ -0,0 +1,29 @@
|
||||
# Asset Processing Utility
|
||||
|
||||
This tool streamlines the conversion of raw 3D asset source files from supplies (archives or folders) into a configurable library format.
|
||||
Goals include automatically updating Assets in various DCC's on import as well - minimising end user workload.
|
||||
|
||||
## Features
|
||||
|
||||
* **Multithreaded Processing:** Leverages multiple CPU cores for parallel input processing, especially beneficial for large batches of inputs.
|
||||
* **Token-based Output Naming:** Offers highly flexible and configurable output directory structures and file naming conventions using a token system (e.g., `[supplier]`, `[assetname]`, `[resolution]`).
|
||||
* **Adjustable Resolutions:** Automatically resizes texture maps to multiple user-defined resolutions (e.g., 4K, 2K, 1K) during processing.
|
||||
* **Adjustable image formats:** Allows configuration of prefered image output formats, allowing for overrides depending on various factors (prefer EXR over PNG for 16Bit, Prefer JPG over PNG for resolutions above certain thesholds). User can configure compression settings for each image format.
|
||||
* **Gloss to Roughness Conversions:** Includes functionality to automatically invert Glossiness maps to Roughness maps, ensuring PBR workflow compatibility.
|
||||
* **Configurable Channel Packing:** Supports customizable channel packing, allowing users to define rules for merging channels from different source maps into optimized packed textures (e.g., Normal + Roughness into a single NRMRGH map).
|
||||
* **User-Friendly GUI:** Provides an intuitive Graphical User Interface for easy drag-and-drop input, interactive review and refinement of asset predictions, preset management, and process monitoring.
|
||||
* **Asset Identification by LLM:** Supports using customizable Large Language Model (LLM) for intelligent and flexible identification of asset and file types.
|
||||
* **Customizable Asset and File Types:** Allows for extensive customization of asset and file type definitions through configurable presets, enabling adaptation to various asset sources and conventions.
|
||||
* **Blender Asset Catalog Post-processing Step:** Optionally runs Blender scripts after processing, providing automatic creation of PBR node groups and materials, and marking them as assets for use in Blender's Asset Browser without needing user interaction.
|
||||
|
||||
## Core Workflow
|
||||
|
||||
1. **Input Sources:** Drop asset archives or folders into the GUI.
|
||||
2. **Identify Inputs:** The tool reads the directory contents and performs an initial identification of asset and file types using configurable presets or an customizable LLM. Users can then review and refine these predictions interactively in a preview table. This involves confirming or correcting asset names, suppliers, types, and individual file types using direct editing, keyboard shortcuts, or triggering re-interpretation for specific sources.
|
||||
3. **Export to Library:** Once the inputs are correctly arranged and user verified, initiate the processing. The tool then automates tasks such as file classification, image resizings, format conversion, channel packing, and metadata generation, exporting the processed assets to a configurable output directory structure and naming convention. Most processing steps can be configured to fit user requirements.
|
||||
|
||||
In addition to the interactive GUI, the tool also offers a Command-Line Interface (CLI) for batch processing and scripting, and a Directory Monitor for automated processing of files dropped into a watched folder. (Docker Support Planned)
|
||||
|
||||
## Documentation
|
||||
|
||||
For detailed information on installation, configuration, usage of the different interfaces, and the output structure, please refer to the [`Documentation`](Documentation/00_Overview.md)
|
||||
Binary file not shown.
57
TestFiles/Test-BoucleChunky001.json
Normal file
57
TestFiles/Test-BoucleChunky001.json
Normal file
@ -0,0 +1,57 @@
|
||||
{
|
||||
"source_rules": [
|
||||
{
|
||||
"input_path": "BoucleChunky001.zip",
|
||||
"supplier_identifier": "Dinesen",
|
||||
"preset_name": null,
|
||||
"assets": [
|
||||
{
|
||||
"asset_name": "BoucleChunky001",
|
||||
"asset_type": "Surface",
|
||||
"files": [
|
||||
{
|
||||
"file_path": "BoucleChunky001_AO_1K_METALNESS.png",
|
||||
"item_type": "MAP_AO",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_COL_1K_METALNESS.png",
|
||||
"item_type": "MAP_COL",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_DISP16_1K_METALNESS.png",
|
||||
"item_type": "MAP_DISP",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_DISP_1K_METALNESS.png",
|
||||
"item_type": "MAP_DISP",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_Fabric.png",
|
||||
"item_type": "EXTRA",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_METALNESS_1K_METALNESS.png",
|
||||
"item_type": "MAP_METAL",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_NRM_1K_METALNESS.png",
|
||||
"item_type": "MAP_NRM",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
},
|
||||
{
|
||||
"file_path": "BoucleChunky001_ROUGHNESS_1K_METALNESS.png",
|
||||
"item_type": "MAP_ROUGH",
|
||||
"target_asset_name_override": "BoucleChunky001"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@ -1,139 +0,0 @@
|
||||
---
|
||||
ID: FEAT-008
|
||||
Type: Feature
|
||||
Status: Backlog
|
||||
Priority: Medium
|
||||
Labels: [gui, enhancement]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related: gui/main_window.py, gui/prediction_handler.py
|
||||
---
|
||||
|
||||
# [FEAT-008]: Refine GUI Preview Table Display and Sorting
|
||||
|
||||
## Description
|
||||
Enhance the usability and clarity of the GUI's detailed file preview table. This involves improving the default sorting to group by asset and then status, removing redundant information, simplifying status text, and rearranging columns.
|
||||
|
||||
## Current Behavior
|
||||
* The preview table lists all files from all assets together without clear grouping.
|
||||
* The table does not sort by status by default, or the sort order isn't optimized for clarity.
|
||||
* The table includes a "Predicted Output" column which is considered redundant.
|
||||
* Status text can be verbose (e.g., "unmatched extra", "[Unmatched Extra (Regex match: #####)]", "Ignored (Superseed by 16bit variant for ####)").
|
||||
* The "Original Path" column is not necessarily the rightmost column.
|
||||
|
||||
## Desired Behavior / Goals
|
||||
* The preview table should group files by the asset they belong to (e.g., based on the input ZIP/folder name or derived asset name).
|
||||
* Within each asset group, the table should sort rows by 'Status' by default.
|
||||
* The default secondary sort order for 'Status' should prioritize actionable or problematic statuses, grouping models with their maps: Error > (Mapped & Model) > Ignored > Extra. (Note: Assumes 'Unrecognised' files are displayed as 'Extra').
|
||||
* Within the 'Mapped & Model' group, files should be sorted alphabetically by original path or filename to keep related items together.
|
||||
* The "Predicted Output" column should be removed from the table view.
|
||||
* Status display text should be made more concise:
|
||||
* "unmatched extra" should be displayed as "Extra".
|
||||
* "[Unmatched Extra (Regex match: #####)]" should be displayed as "[Extra=#####]".
|
||||
* "Ignored (Superseed by 16bit variant for ####)" should be displayed as "Superseeded by 16bit ####".
|
||||
* Other statuses ("Mapped", "Model", "Error") should remain clear.
|
||||
* The "Original Path" column should be positioned as the rightmost column in the table.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
* Modifications will likely be needed in `gui/main_window.py` (table view setup, column management, data population, sorting proxy model) and `gui/prediction_handler.py` (ensure prediction results include the source asset identifier).
|
||||
* The table model (`QAbstractTableModel` or similar) needs to store the source asset identifier for each file row.
|
||||
* Implement a custom multi-level sorting logic using `QSortFilterProxyModel`.
|
||||
* The primary sort key will be the asset identifier.
|
||||
* The secondary sort key will be the status, mapping 'Mapped' and 'Model' to the same priority level.
|
||||
* The tertiary sort key will be the original path/filename.
|
||||
* Update the column hiding/showing logic to remove the "Predicted Output" column.
|
||||
* Implement string formatting or replacement logic for the status display text.
|
||||
* Adjust the column order.
|
||||
* Consider the performance implications of grouping and multi-level sorting, especially with a large number of assets/files. Add necessary optimizations (e.g., efficient data structures, potentially deferring sorting until needed).
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [ ] When assets are added to the queue and detailed preview is active, the table visually groups files by their source asset.
|
||||
* [ ] Within each asset group, the table automatically sorts by the Status column according to the specified multi-level order.
|
||||
* [ ] Clicking the Status header cycles through ascending/descending sort based on the custom multi-level order: Asset > Status (Error > (Mapped & Model) > Ignored > Extra) > Filename.
|
||||
* [ ] The "Predicted Output" column is not present in the detailed preview table.
|
||||
* [ ] Status text for relevant items displays concisely as: "Extra", "[Extra=#####]", "Superseeded by 16bit ####".
|
||||
* [ ] The "Original Path" column is visually the last column on the right side of the table.
|
||||
---
|
||||
|
||||
## Implementation Plan (Generated 2025-04-22)
|
||||
|
||||
This plan outlines the steps to implement the GUI preview refinements described in this ticket.
|
||||
|
||||
**Goal:** Enhance the GUI's detailed file preview table for better usability and clarity by implementing grouping, refined sorting, simplified status text, and adjusted column layout.
|
||||
|
||||
**Affected Files:**
|
||||
|
||||
* `gui/main_window.py`: Handles table view setup, data population trigger, and column management.
|
||||
* `gui/preview_table_model.py`: Contains the table model (`PreviewTableModel`) and the sorting proxy model (`PreviewSortFilterProxyModel`).
|
||||
|
||||
**Plan Steps:**
|
||||
|
||||
1. **Correct Data Population in `main_window.py`:**
|
||||
* **Action:** Modify the `on_prediction_results_ready` slot (around line 893).
|
||||
* **Change:** Update the code to call `self.preview_model.set_data(results)` instead of populating the old `self.preview_table`.
|
||||
* **Remove:** Delete code manually setting headers, rows, and items on `self.preview_table`.
|
||||
* **Keep:** Retain the initial setup of `self.preview_table_view` (lines 400-424).
|
||||
|
||||
2. **Implement Status Text Simplification in `preview_table_model.py`:**
|
||||
* **Action:** Modify the `data()` method within the `PreviewTableModel` class (around line 56).
|
||||
* **Change:** Inside the `if role == Qt.ItemDataRole.DisplayRole:` block for `COL_STATUS`, add logic to transform the raw status string:
|
||||
* "Unmatched Extra" -> "Extra"
|
||||
* "[Unmatched Extra (Regex match: PATTERN)]" -> "[Extra=PATTERN]"
|
||||
* "Ignored (Superseed by 16bit variant for FILENAME)" -> "Superseeded by 16bit FILENAME"
|
||||
* Otherwise, return original status.
|
||||
* **Note:** Ensure `ROLE_RAW_STATUS` still returns the original status for sorting.
|
||||
|
||||
3. **Refine Sorting Logic in `preview_table_model.py`:**
|
||||
* **Action:** Modify the `lessThan` method within the `PreviewSortFilterProxyModel` class (around line 166).
|
||||
* **Change:** Adjust the "Level 2: Sort by Status" logic to use a priority mapping where "Mapped" and "Model" have the same priority index, causing the sort to fall through to Level 3 (Path) for items within this group.
|
||||
```python
|
||||
# Example Priority Mapping
|
||||
STATUS_PRIORITY = {
|
||||
"Error": 0,
|
||||
"Mapped": 1,
|
||||
"Model": 1,
|
||||
"Ignored": 2,
|
||||
"Extra": 3,
|
||||
"Unrecognised": 3,
|
||||
"Unmatched Extra": 3,
|
||||
"[No Status]": 99
|
||||
}
|
||||
# ... comparison logic using STATUS_PRIORITY ...
|
||||
```
|
||||
* **Remove/Update:** Replace the old `STATUS_ORDER` list with this priority dictionary logic.
|
||||
|
||||
4. **Adjust Column Order in `main_window.py`:**
|
||||
* **Action:** Modify the `setup_main_panel_ui` method (around lines 404-414).
|
||||
* **Change:** After setting up `self.preview_table_view`, explicitly move the "Original Path" column to the last visual position using `header.moveSection()`.
|
||||
* **Verify:** Ensure the "Predicted Output" column remains hidden.
|
||||
|
||||
**Visual Plan (Mermaid):**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Start FEAT-008 Implementation] --> B(Refactor `main_window.py::on_prediction_results_ready`);
|
||||
B --> C{Use `self.preview_model.set_data()`?};
|
||||
C -- Yes --> D(Remove manual `QTableWidget` population);
|
||||
C -- No --> E[ERROR: Incorrect data flow];
|
||||
D --> F(Modify `preview_table_model.py::PreviewTableModel::data()`);
|
||||
F --> G{Implement Status Text Simplification?};
|
||||
G -- Yes --> H(Add formatting logic for DisplayRole);
|
||||
G -- No --> I[ERROR: Status text not simplified];
|
||||
H --> J(Modify `preview_table_model.py::PreviewSortFilterProxyModel::lessThan()`);
|
||||
J --> K{Implement Mapped/Model Grouping & Custom Sort Order?};
|
||||
K -- Yes --> L(Use priority mapping for status comparison);
|
||||
K -- No --> M[ERROR: Sorting incorrect];
|
||||
L --> N(Modify `main_window.py::setup_main_panel_ui()`);
|
||||
N --> O{Move 'Original Path' Column to End?};
|
||||
O -- Yes --> P(Use `header.moveSection()`);
|
||||
O -- No --> Q[ERROR: Column order incorrect];
|
||||
P --> R(Verify All Acceptance Criteria);
|
||||
R --> S[End FEAT-008 Implementation];
|
||||
|
||||
subgraph main_window.py Modifications
|
||||
B; D; N; P;
|
||||
end
|
||||
|
||||
subgraph preview_table_model.py Modifications
|
||||
F; H; J; L;
|
||||
end
|
||||
@ -1,76 +0,0 @@
|
||||
# FEAT-009: GUI - Unify Preset Editor Selection and Processing Preset
|
||||
|
||||
**Status:** Resolved
|
||||
**Priority:** Medium
|
||||
**Assigned:** TBD
|
||||
**Reporter:** Roo (Architect Mode)
|
||||
**Date:** 2025-04-22
|
||||
|
||||
## Description
|
||||
|
||||
This ticket proposes a GUI modification to streamline the preset handling workflow by unifying the preset selection mechanism. Currently, the preset selected for editing in the left panel can be different from the preset selected for processing/previewing in the right panel. This change aims to eliminate this duality, making the preset loaded in the editor the single source of truth for both editing and processing actions.
|
||||
|
||||
## Current Behavior
|
||||
|
||||
1. **Preset Editor Panel (Left):** Users select a preset from the 'Preset List'. This action loads the selected preset's details into the 'Preset Editor Tabs' for viewing or modification (See `readme.md` lines 257-260).
|
||||
2. **Processing Panel (Right):** A separate 'Preset Selector' dropdown exists. Users select a preset from this dropdown *specifically* for processing the queued assets and generating the file preview in the 'Preview Table' (See `readme.md` lines 262, 266).
|
||||
3. This allows for a scenario where the preset being edited is different from the preset used for previewing and processing, potentially causing confusion.
|
||||
|
||||
## Proposed Behavior
|
||||
|
||||
1. **Unified Selection:** Selecting a preset from the 'Preset List' in the left panel will immediately make it the active preset for *both* editing (loading its details into the 'Preset Editor Tabs') *and* processing/previewing.
|
||||
2. **Dropdown Removal:** The separate 'Preset Selector' dropdown in the right 'Processing Panel' will be removed.
|
||||
3. **Dynamic Updates:** The 'Preview Table' in the right panel will dynamically update based on the preset selected in the left panel's 'Preset List'.
|
||||
4. **Processing Logic:** The 'Start Processing' action will use the currently active preset (selected from the left panel's list).
|
||||
|
||||
## Rationale
|
||||
|
||||
* **Improved User Experience:** Simplifies the UI by removing a redundant control and creates a more intuitive workflow.
|
||||
* **Reduced Complexity:** Eliminates the need to manage two separate preset states (editing vs. processing).
|
||||
* **Consistency:** Ensures that the preview and processing actions always reflect the preset currently being viewed/edited.
|
||||
|
||||
## Implementation Notes/Tasks
|
||||
|
||||
* **UI Modification (`gui/main_window.py`):**
|
||||
* Remove the `QComboBox` (or similar widget) used for the 'Preset Selector' from the right 'Processing Panel' layout.
|
||||
* **Signal/Slot Connection (`gui/main_window.py`):**
|
||||
* Ensure the signal emitted when a preset is selected in the left panel's 'Preset List' (`QListWidget` or similar) is connected to slots responsible for:
|
||||
* Loading the preset into the editor tabs.
|
||||
* Updating the application's state to reflect the new active preset for processing.
|
||||
* Triggering an update of the 'Preview Table'.
|
||||
* **State Management:**
|
||||
* Modify how the currently active preset for processing is stored and accessed. It should now directly reference the preset selected in the left panel list.
|
||||
* **Handler Updates (`gui/prediction_handler.py`, `gui/processing_handler.py`):**
|
||||
* Ensure these handlers correctly retrieve and use the single, unified active preset state when generating previews or initiating processing.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. The 'Preset Selector' dropdown is no longer visible in the right 'Processing Panel'.
|
||||
2. Selecting a preset in the 'Preset List' (left panel) successfully loads its details into the 'Preset Editor Tabs'.
|
||||
3. Selecting a preset in the 'Preset List' (left panel) updates the 'Preview Table' (right panel) to reflect the rules of the newly selected preset.
|
||||
4. Initiating processing via the 'Start Processing' button uses the preset currently selected in the 'Preset List' (left panel).
|
||||
5. The application remains stable and performs processing correctly with the unified preset selection.
|
||||
|
||||
## Workflow Diagram (Mermaid)
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Current GUI"
|
||||
A[Preset List (Left Panel)] -- Selects --> B(Preset Editor Tabs);
|
||||
C[Preset Selector Dropdown (Right Panel)] -- Selects --> D(Active Processing Preset);
|
||||
D -- Affects --> E(Preview Table);
|
||||
D -- Used by --> F(Processing Logic);
|
||||
end
|
||||
|
||||
subgraph "Proposed GUI"
|
||||
G[Preset List (Left Panel)] -- Selects --> H(Preset Editor Tabs);
|
||||
G -- Also Sets --> I(Active Preset for Editing & Processing);
|
||||
I -- Affects --> J(Preview Table);
|
||||
I -- Used by --> K(Processing Logic);
|
||||
L(Preset Selector Dropdown Removed);
|
||||
end
|
||||
|
||||
A --> G;
|
||||
B --> H;
|
||||
E --> J;
|
||||
F --> K;
|
||||
@ -1,49 +0,0 @@
|
||||
# FEAT-GUI-NoDefaultPreset: Prevent Default Preset Selection in GUI
|
||||
|
||||
## Objective
|
||||
|
||||
Modify the Graphical User Interface (GUI) to prevent any preset from being selected by default on application startup. Instead, the GUI should prompt the user to explicitly select a preset from the list before displaying the detailed file preview. This aims to avoid accidental processing with an unintended preset.
|
||||
|
||||
## Problem Description
|
||||
|
||||
Currently, when the GUI application starts, a preset from the available list is automatically selected. This can lead to user confusion if they are not aware of this behavior and proceed to add assets and process them using a preset they did not intend to use. The preview table also populates automatically based on this default selection, which might not be desired until a conscious preset choice is made.
|
||||
|
||||
## Failed Attempts
|
||||
|
||||
1. **Attempt 1: Remove Default Selection Logic and Add Placeholder Text:**
|
||||
* **Approach:** Removed code that explicitly set a default selected item in the preset list during initialization. Added a `QLabel` with placeholder text ("Please select a preset...") to the preview area and attempted to use `setPlaceholderText` on the `QTableView` (this was incorrect as `QTableView` does not have this method). Managed visibility of the placeholder label and table view.
|
||||
* **Result:** The `setPlaceholderText` call failed with an `AttributeError`. Even after removing the erroneous line and adding a dedicated `QLabel`, a preset was still being selected automatically in the list on startup, and the placeholder was not consistently shown. This suggested that simply populating the `QListWidget` might implicitly trigger a selection.
|
||||
|
||||
2. **Attempt 2: Explicitly Clear Selection and Refine Visibility Logic:**
|
||||
* **Approach:** Added explicit calls (`setCurrentItem(None)`, `clearSelection()`) after populating the preset list to ensure no item was selected. Refined the visibility logic for the placeholder label and table view in `_clear_editor` and `_load_selected_preset_for_editing`. Added logging to track selection and visibility changes.
|
||||
* **Result:** Despite explicitly clearing the selection, testing indicated that a preset was still being selected on startup, and the placeholder was not consistently displayed. This reinforced the suspicion that the `QListWidget`'s behavior upon population was automatically triggering a selection and the associated signal.
|
||||
|
||||
## Proposed Plan: Implement "-- Select a Preset --" Placeholder Item
|
||||
|
||||
This approach makes the "no selection" state an explicit, selectable item in the preset list, giving us more direct control over the initial state and subsequent behavior.
|
||||
|
||||
1. **Modify `populate_presets` Method:**
|
||||
* Add a `QListWidgetItem` with the text "-- Select a Preset --" at the very beginning of the list (index 0) after clearing the list but before adding actual preset items.
|
||||
* Store a special, non-`Path` value (e.g., `None` or a unique string like `"__PLACEHOLDER__"`) in this placeholder item's `UserRole` data to distinguish it from real presets.
|
||||
* After adding all real preset items, explicitly set the current item to this placeholder item using `self.editor_preset_list.setCurrentRow(0)`.
|
||||
|
||||
2. **Modify `_load_selected_preset_for_editing` Method (Slot for `currentItemChanged`):**
|
||||
* At the beginning of the method, check if the `current_item` is the placeholder item by examining its `UserRole` data.
|
||||
* If the placeholder item is selected:
|
||||
* Call `self._clear_editor()` to reset all editor fields.
|
||||
* Call `self.preview_model.clear_data()` to ensure the preview table model is empty.
|
||||
* Explicitly set `self.preview_placeholder_label.setVisible(True)` and `self.preview_table_view.setVisible(False)`.
|
||||
* Return from the method without proceeding to load a preset or call `update_preview`.
|
||||
* If a real preset item is selected, proceed with the existing logic: get the `Path` from the item's data, call `_load_preset_for_editing(preset_path)`, call `self.update_preview()`, set `self.preview_placeholder_label.setVisible(False)` and `self.preview_table_view.setVisible(True)`.
|
||||
|
||||
3. **Modify `start_processing` Method:**
|
||||
* Before initiating the processing, check if the currently selected item in `editor_preset_list` is the placeholder item.
|
||||
* If the placeholder item is selected, display a warning message to the user (e.g., "Please select a valid preset before processing.") using the status bar and return from the method.
|
||||
* If a real preset is selected, proceed with the existing processing logic.
|
||||
|
||||
4. **Modify `update_preview` Method:**
|
||||
* Add a check at the beginning of the method. Get the `current_item` from `editor_preset_list`. If it is the placeholder item, clear the preview model (`self.preview_model.clear_data()`) and return immediately. This prevents the prediction handler from running when no valid preset is selected.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Implement the proposed plan by modifying the specified methods in `gui/main_window.py`. Test the GUI on startup and when selecting different items in the preset list to ensure the desired behavior is achieved.
|
||||
@ -1,38 +0,0 @@
|
||||
---
|
||||
ID: ISSUE-004
|
||||
Type: Issue
|
||||
Status: Resolved
|
||||
Priority: High
|
||||
Labels: [bug, core, image-processing]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related:
|
||||
---
|
||||
|
||||
# [ISSUE-004]: Color channel swapping in image processing (Normal Maps, Stats)
|
||||
|
||||
## Description
|
||||
There appears to be a general issue with how color channels (specifically Red and Blue) are handled in several parts of the image processing pipeline. This has been observed in normal map channel packing and potentially in the calculation of image statistics per channel, where the Red and Blue channels seem to be swapped relative to their intended order.
|
||||
|
||||
## Current Behavior
|
||||
When processing images where individual color channels are accessed or manipulated (e.g., during normal map channel packing or calculating per-channel image statistics), the Red and Blue channels appear to be swapped. For example, in normal map packing, the channel intended for Blue might contain Red data, and vice versa, while Green remains correct. This suggests a consistent R/B channel inversion issue in the underlying image processing logic.
|
||||
|
||||
## Desired Behavior / Goals
|
||||
The tool should correctly handle color channels according to standard RGB ordering in all image processing operations, including channel packing and image statistics calculation. The Red, Green, and Blue channels should consistently correspond to their intended data.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
(This likely points to an issue in the image loading, channel splitting, merging, or processing functions, possibly related to the library used for image manipulation (e.g., OpenCV). Need to investigate how channels are accessed and ordered in relevant code sections like `_process_maps`, `_merge_maps`, and image statistics calculation.)
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [ ] Process an asset with a normal map and a map requiring channel packing (e.g., NRMRGH).
|
||||
* [ ] Verify that the channels in the output normal map and packed map are in the correct R, G, B order.
|
||||
* [ ] Verify that the calculated image statistics (Min/Max/Mean) for Red and Blue channels accurately reflect the data in those specific channels, not the swapped data.
|
||||
## Resolution
|
||||
The root cause was identified as a mismatch between OpenCV's default BGR channel order upon image loading (`cv2.imread`) and subsequent code assuming an RGB channel order, particularly in channel indexing during map merging (`_merge_maps`) and potentially in statistics calculation (`_calculate_image_stats`).
|
||||
|
||||
The fix involved the following changes in `asset_processor.py`:
|
||||
1. **BGR to RGB Conversion:** Immediately after loading a 3-channel image using `cv2.imread` in the `_process_maps` function, the image is converted to RGB color space using `img_processed = cv2.cvtColor(img_loaded, cv2.COLOR_BGR2RGB)`. Grayscale or 4-channel images are handled appropriately without conversion.
|
||||
2. **Updated Channel Indexing:** The channel indexing logic within the `_merge_maps` function was updated to reflect the RGB order (Red = index 0, Green = index 1, Blue = index 2) when extracting channels from the now-RGB source images for merging.
|
||||
3. **Statistics Assumption Update:** Comments in `_calculate_image_stats` were updated to reflect that the input data is now expected in RGB order.
|
||||
|
||||
This ensures consistent RGB channel ordering throughout the processing pipeline after the initial load.
|
||||
@ -1,83 +0,0 @@
|
||||
---
|
||||
ID: ISSUE-010
|
||||
Type: Issue
|
||||
Status: Resolved
|
||||
Priority: High
|
||||
Labels: [bug, core, image-processing, regression, resolved]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related: #ISSUE-004, asset_processor.py
|
||||
---
|
||||
|
||||
# [ISSUE-010]: Color Channel Swapping Regression for RGB/Merged Maps after ISSUE-004 Fix
|
||||
|
||||
## Description
|
||||
The resolution implemented for `ISSUE-004` successfully corrected the BGR/RGB channel handling for image statistics calculation and potentially normal map packing. However, this fix appears to have introduced a regression where standard RGB color maps and maps generated through merging operations are now being saved with swapped Blue and Red channels (BGR order) instead of the expected RGB order.
|
||||
|
||||
## Current Behavior
|
||||
- Standard RGB texture maps (e.g., Diffuse, Albedo) loaded and processed are saved with BGR channel order.
|
||||
- Texture maps created by merging channels (e.g., ORM, NRMRGH) are saved with BGR channel order for their color components.
|
||||
- Image statistics (`metadata.json`) are calculated correctly based on RGB data.
|
||||
- Grayscale maps are handled correctly.
|
||||
- RGBA maps are assumed to be handled correctly (needs verification).
|
||||
|
||||
## Desired Behavior / Goals
|
||||
- All processed color texture maps (both original RGB and merged maps) should be saved with the standard RGB channel order.
|
||||
- Image statistics should continue to be calculated correctly based on RGB data.
|
||||
- Grayscale and RGBA maps should retain their correct handling.
|
||||
- The fix should not reintroduce the problems solved by `ISSUE-004`.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
The issue likely stems from the universal application of `cv2.cvtColor(img_loaded, cv2.COLOR_BGR2RGB)` in `_process_maps` introduced in the `ISSUE-004` fix. While this ensures consistent RGB data for internal operations like statistics and merging (which now expects RGB), the final saving step (`cv2.imwrite`) might implicitly expect BGR data for color images, or the conversion needs to be selectively undone before saving certain map types.
|
||||
|
||||
Investigation needed in `asset_processor.py`:
|
||||
- Review `_process_maps`: Where is the BGR->RGB conversion happening? Is it applied to all 3-channel images?
|
||||
- Review `_process_maps` saving logic: How are different map types saved? Does `cv2.imwrite` expect RGB or BGR?
|
||||
- Review `_merge_maps`: How are channels combined? Does the saving logic here also need adjustment?
|
||||
- Determine if the BGR->RGB conversion should be conditional or if a final RGB->BGR conversion is needed before saving specific map types.
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [x] Process an asset containing standard RGB maps (e.g., Color/Albedo). Verify the output map has correct RGB channel order.
|
||||
* [x] Process an asset requiring map merging (e.g., ORM from Roughness, Metallic, AO). Verify the output merged map has correct RGB(A) channel order.
|
||||
* [x] Verify image statistics in `metadata.json` remain correct (reflecting RGB values).
|
||||
* [x] Verify grayscale maps are processed and saved correctly.
|
||||
* [x] Verify normal maps are processed and saved correctly (as per ISSUE-004 fix).
|
||||
* [ ] (Optional) Verify RGBA maps are processed and saved correctly.
|
||||
---
|
||||
|
||||
## Resolution
|
||||
The issue was resolved by implementing conditional RGB to BGR conversion immediately before saving images using `cv2.imwrite` within the `_process_maps` and `_merge_maps` methods in `asset_processor.py`.
|
||||
|
||||
The logic checks if the image is 3-channel and if the target output format is *not* EXR. If both conditions are true, the image data is converted from the internal RGB representation back to BGR, which is the expected channel order for `cv2.imwrite` when saving formats like PNG, JPG, and TIF.
|
||||
|
||||
This approach ensures that color maps are saved with the correct channel order in standard formats while leaving EXR files (which handle RGB correctly) and grayscale/single-channel images unaffected. It also preserves the internal RGB representation used for accurate image statistics calculation and channel merging, thus not reintroducing the issues fixed by `ISSUE-004`.
|
||||
|
||||
## Implementation Plan (Generated 2025-04-22)
|
||||
|
||||
**Goal:** Correct the BGR/RGB channel order regression for saved color maps (introduced by the `ISSUE-004` fix) while maintaining the correct handling for statistics, grayscale maps, and EXR files.
|
||||
|
||||
**Core Idea:** Convert the image data back from RGB to BGR *just before* saving with `cv2.imwrite`, but only for 3-channel images and formats where this conversion is necessary (e.g., PNG, JPG, TIF, but *not* EXR).
|
||||
|
||||
**Plan Steps:**
|
||||
|
||||
1. **Modify `_process_maps` Saving Logic (`asset_processor.py`):**
|
||||
* Before the primary `cv2.imwrite` call (around line 1182) and the fallback call (around line 1206), add logic to check if the image is 3-channel and the output format is *not* 'exr'. If true, convert the image from RGB to BGR using `cv2.cvtColor` and add a debug log message.
|
||||
2. **Modify `_merge_maps` Saving Logic (`asset_processor.py`):**
|
||||
* Apply the same conditional RGB to BGR conversion logic before the primary `cv2.imwrite` call (around line 1505) and the fallback call (around line 1531), including a debug log message.
|
||||
3. **Verification Strategy:**
|
||||
* Test with various 3-channel color maps (Albedo, Emission, etc.) and merged maps (NRMRGH, etc.) saved as PNG/JPG/TIF.
|
||||
* Verify correct RGB order in outputs.
|
||||
* Verify grayscale, EXR, and statistics calculation remain correct.
|
||||
|
||||
**Mermaid Diagram:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "_process_maps / _merge_maps Saving Step"
|
||||
A[Prepare Final Image Data (in RGB)] --> B{Is it 3-Channel?};
|
||||
B -- Yes --> C{Output Format != 'exr'?};
|
||||
B -- No --> E[Save Image using cv2.imwrite];
|
||||
C -- Yes --> D(Convert Image RGB -> BGR);
|
||||
C -- No --> E;
|
||||
D --> E;
|
||||
end
|
||||
@ -1,29 +0,0 @@
|
||||
---
|
||||
ID: ISSUE-011
|
||||
Type: Issue
|
||||
Status: Backlog
|
||||
Priority: Medium
|
||||
Labels: [blender, bug]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related:
|
||||
---
|
||||
|
||||
# [ISSUE-011]: Blender nodegroup script creates empty assets for skipped items
|
||||
|
||||
## Description
|
||||
The Blender nodegroup creation script (`blenderscripts/create_nodegroups.py`) incorrectly generates empty asset entries in the target .blend file for assets that were skipped during the main processing pipeline. This occurs even though the main asset processor correctly identifies and skips these assets based on the overwrite flag.
|
||||
|
||||
## Current Behavior
|
||||
When running the asset processor with the `--overwrite` flag set to false, if an asset's output directory and metadata.json already exist, the main processing pipeline correctly skips processing that asset. However, the subsequent Blender nodegroup creation script still attempts to create a nodegroup for this skipped asset, resulting in an empty or incomplete asset entry in the target .blend file.
|
||||
|
||||
## Desired Behavior / Goals
|
||||
The Blender nodegroup creation script should only attempt to create nodegroups for assets that were *actually processed* by the main asset processor, not those that were skipped. It should check if the asset was processed successfully before attempting nodegroup creation.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
The `main.py` script, which orchestrates the processing and calls the Blender scripts, needs to pass information about which assets were successfully processed to the Blender nodegroup script. The Blender script should then filter its operations based on this information.
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [ ] Running the asset processor with `--overwrite` false on inputs that already have processed outputs should result in the main processing skipping the asset.
|
||||
* [ ] The Blender nodegroup script, when run after a skipped processing run, should *not* create or update a nodegroup for the skipped asset.
|
||||
* [ ] Only assets that were fully processed should have corresponding nodegroups created/updated by the Blender script.
|
||||
@ -1,29 +0,0 @@
|
||||
---
|
||||
ID: ISSUE-012
|
||||
Type: Issue
|
||||
Status: Backlog
|
||||
Priority: High
|
||||
Labels: [core, bug, image processing]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related:
|
||||
---
|
||||
|
||||
# [ISSUE-012]: MASK map processing fails to extract alpha channel from RGBA images
|
||||
|
||||
## Description
|
||||
When processing texture sets that include MASK maps provided as RGBA images, the asset processor is expected to extract the alpha channel to represent the mask. However, the current implementation appears to be converting the RGBA image to grayscale instead of isolating the alpha channel. This results in incorrect MASK maps in the processed output.
|
||||
|
||||
## Current Behavior
|
||||
The asset processor, when encountering an RGBA image classified as a MASK map, converts the image to grayscale. The alpha channel information is lost or ignored during this process.
|
||||
|
||||
## Desired Behavior / Goals
|
||||
For RGBA images classified as MASK maps, the asset processor should extract the alpha channel and use it as the content for the output MASK map. The resulting MASK map should be a single-channel (grayscale) image representing the transparency or mask information from the original alpha channel.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
The image processing logic within `asset_processor.py` (likely in the `_process_maps` method or a related helper function) needs to be updated to specifically handle RGBA input images for MASK map types. It should identify the alpha channel and save it as a single-channel output image. Libraries like OpenCV or Pillow should provide functionality for accessing individual channels.
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [ ] Process an asset with an RGBA image designated as a MASK map according to the preset.
|
||||
* [ ] Verify that the output MASK map is a single-channel grayscale image.
|
||||
* [ ] Confirm that the grayscale values in the output MASK map correspond to the alpha values of the original RGBA input image.
|
||||
@ -1,53 +0,0 @@
|
||||
# Plan for ISSUE-012: MASK map processing fails to extract alpha channel from RGBA images
|
||||
|
||||
## Issue Description
|
||||
When processing texture sets that include MASK maps provided as RGBA images, the asset processor is expected to extract the alpha channel to represent the mask. However, the current implementation appears to be converting the RGBA image to grayscale instead of isolating the alpha channel, resulting in incorrect MASK maps. This issue affects all RGBA images classified as MASK, including plain 'MASK' and MASK variants (e.g., MASK-1).
|
||||
|
||||
## Analysis
|
||||
Based on the code in `asset_processor.py`, specifically the `_load_and_transform_source` method, the issue likely stems from the order of operations. The current logic converts 4-channel BGRA images to 3-channel RGB *before* checking specifically for MASK types and attempting to extract the alpha channel. This means the alpha channel is lost before the code gets a chance to extract it. The condition `if map_type.upper() == 'MASK':` is too strict and does not cover MASK variants.
|
||||
|
||||
## Detailed Plan
|
||||
|
||||
1. **Analyze `_load_and_transform_source`:** Re-examine the `_load_and_transform_source` method in `asset_processor.py` to confirm the exact sequence of image loading, color space conversions, and MASK-specific handling. (Completed during initial analysis).
|
||||
2. **Modify MASK Handling Condition:** Change the current condition `if map_type.upper() == 'MASK':` to use the `_get_base_map_type` helper function, so it correctly identifies all MASK variants (e.g., 'MASK', 'MASK-1', 'MASK-2') and applies the special handling logic to them. The condition will become `if _get_base_map_type(map_type) == 'MASK':`.
|
||||
3. **Reorder and Refine Logic:**
|
||||
* The image will still be loaded with `cv2.IMREAD_UNCHANGED` for MASK types to ensure the alpha channel is initially present.
|
||||
* Immediately after loading, and *before* any general color space conversions (like BGR->RGB), check if the base map type is 'MASK' and if the loaded image is 4-channel (RGBA/BGRA).
|
||||
* If both conditions are true, extract the alpha channel (`img_loaded[:, :, 3]`) and use this single-channel data for subsequent processing steps (`img_prepared`).
|
||||
* If the base map type is 'MASK' but the loaded image is 3-channel (RGB/BGR), convert it to grayscale (`cv2.cvtColor(img_loaded, cv2.COLOR_BGR2GRAY)`) and use this for `img_prepared`.
|
||||
* If the base map type is 'MASK' and the loaded image is already 1-channel (grayscale), keep it as is.
|
||||
* If the base map type is *not* 'MASK', the existing BGR->RGB conversion logic for 3/4 channel images will be applied as before.
|
||||
4. **Review Subsequent Steps:** Verify that the rest of the `_load_and_transform_source` method (Gloss->Rough inversion, resizing, dtype conversion) correctly handles the single-channel image data that will now be produced for RGBA MASK inputs.
|
||||
5. **Testing:** Use the acceptance criteria outlined in `ISSUE-012` to ensure the fix works correctly.
|
||||
|
||||
## Proposed Code Logic Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Load Image (IMREAD_UNCHANGED for MASK)] --> B{Loading Successful?};
|
||||
B -- No --> C[Handle Load Error];
|
||||
B -- Yes --> D[Get Original Dtype & Shape];
|
||||
D --> E{Base Map Type is MASK?};
|
||||
E -- Yes --> F{Loaded Image is 4-Channel?};
|
||||
F -- Yes --> G[Extract Alpha Channel];
|
||||
F -- No --> H{Loaded Image is 3-Channel?};
|
||||
H -- Yes --> I[Convert BGR/RGB to Grayscale];
|
||||
H -- No --> J[Keep as is (Assume Grayscale)];
|
||||
G --> K[Set img_prepared to Mask Data];
|
||||
I --> K;
|
||||
J --> K;
|
||||
E -- No --> L{Loaded Image is 3 or 4-Channel?};
|
||||
L -- Yes --> M[Convert BGR/BGRA to RGB];
|
||||
L -- No --> N[Keep as is];
|
||||
M --> O[Set img_prepared to RGB Data];
|
||||
N --> O;
|
||||
K --> P[Proceed with other transformations (Gloss, Resize, etc.)];
|
||||
O --> P;
|
||||
P --> Q[Cache Result & Return];
|
||||
C --> Q;
|
||||
```
|
||||
|
||||
## Acceptance Criteria (from ISSUE-012)
|
||||
* [ ] Process an asset with an RGBA image designated as a MASK map according to the preset.
|
||||
* [ ] Verify that the output MASK map is a single-channel grayscale image.
|
||||
* [ ] Confirm that the grayscale values in the output MASK map correspond to the alpha values of the original RGBA input image.
|
||||
@ -1,30 +0,0 @@
|
||||
---
|
||||
ID: ISSUE-013
|
||||
Type: Issue
|
||||
Status: Backlog
|
||||
Priority: Medium
|
||||
Labels: [core, bug, refactor, image processing]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22
|
||||
Related: #REFACTOR-001-merge-from-source
|
||||
|
||||
---
|
||||
|
||||
# [ISSUE-013]: Image statistics calculation missing for roughness maps after merged map refactor
|
||||
|
||||
## Description
|
||||
Following the recent refactor of the merged map processing logic, the calculation of image statistics (Min/Max/Mean) for roughness maps appears to have been omitted or broken. This data is valuable for quality control and metadata, and needs to be reimplemented for roughness maps, particularly when they are part of a merged texture like NRMRGH.
|
||||
|
||||
## Current Behavior
|
||||
Image statistics (Min/Max/Mean) are not being calculated or stored for roughness maps after the merged map refactor.
|
||||
|
||||
## Desired Behavior / Goals
|
||||
Reimplement the image statistics calculation for roughness maps. Ensure that statistics are calculated correctly for standalone roughness maps and for roughness data when it is part of a merged map (e.g., the green channel in an NRMRGH map). The calculated statistics should be included in the `metadata.json` file for the asset.
|
||||
|
||||
## Implementation Notes (Optional)
|
||||
Review the changes made during the merged map refactor (`REFACTOR-001-merge-from-source`). Identify where the image statistics calculation for roughness maps was previously handled and integrate it into the new merged map processing flow. Special consideration may be needed to extract the correct channel (green for NRMRGH) for calculation. The `_process_maps` and `_merge_maps` methods in `asset_processor.py` are likely relevant areas.
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
* [ ] Process an asset that includes a roughness map (standalone or as part of a merged map).
|
||||
* [ ] Verify that Min/Max/Mean statistics for the roughness data are calculated.
|
||||
* [ ] Confirm that the calculated roughness statistics are present and correct in the output `metadata.json` file.
|
||||
@ -1,67 +0,0 @@
|
||||
# Plan to Resolve ISSUE-013: Merged Map Roughness Statistics
|
||||
|
||||
**Objective:** Reimplement the calculation and inclusion of image statistics (Min/Max/Mean) for roughness data, covering both standalone roughness maps and roughness data used as a source channel in merged maps (specifically the green channel for NRMRGH), and ensure these statistics are present in the output `metadata.json` file.
|
||||
|
||||
**Proposed Changes:**
|
||||
|
||||
1. **Modify `_merge_maps_from_source`:**
|
||||
* Within the loop that processes each resolution for a merge rule, after successfully loading and transforming the source images into `loaded_inputs_data`, iterate through the `inputs_mapping` for the current rule.
|
||||
* Identify if any of the source map types in the `inputs_mapping` is 'ROUGH'.
|
||||
* If 'ROUGH' is used as a source, retrieve the corresponding loaded image data from `loaded_inputs_data`.
|
||||
* Calculate image statistics (Min/Max/Mean) for this ROUGH source image data using the existing `_calculate_image_stats` helper function.
|
||||
* Store these calculated statistics temporarily, associated with the output merged map type (e.g., 'NRMRGH') and the target channel it's mapped to (e.g., 'G').
|
||||
|
||||
2. **Update Asset Metadata Structure:**
|
||||
* Introduce a new key in the asset's metadata dictionary (e.g., `merged_map_channel_stats`) to store statistics for specific channels within merged maps. This structure will hold the stats per merged map type, per channel, and per resolution (specifically the stats resolution defined in the config).
|
||||
|
||||
3. **Modify `_generate_metadata_file`:**
|
||||
* When generating the final `metadata.json` file, retrieve the accumulated `merged_map_channel_stats` from the asset's metadata dictionary.
|
||||
* Include these statistics under a dedicated section in the output JSON, alongside the existing `image_stats_1k` for individual maps.
|
||||
|
||||
**Data Flow for Statistics Calculation:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Source Image Files] --> B{_load_and_transform_source};
|
||||
B --> C[Resized/Transformed Image Data (float32)];
|
||||
C --> D{Is this source for a ROUGH map?};
|
||||
D -- Yes --> E{_calculate_image_stats};
|
||||
E --> F[Calculated Stats (Min/Max/Mean)];
|
||||
F --> G[Store in merged_map_channel_stats (in asset metadata)];
|
||||
C --> H{_merge_maps_from_source};
|
||||
H --> I[Merged Image Data];
|
||||
I --> J[_save_image];
|
||||
J --> K[Saved Merged Map File];
|
||||
G --> L[_generate_metadata_file];
|
||||
L --> M[metadata.json];
|
||||
|
||||
subgraph Individual Map Processing
|
||||
A --> B;
|
||||
B --> C;
|
||||
C --> N{Is this an individual map?};
|
||||
N -- Yes --> E;
|
||||
E --> O[Store in image_stats_1k (in asset metadata)];
|
||||
C --> P[_save_image];
|
||||
P --> Q[Saved Individual Map File];
|
||||
O --> L;
|
||||
Q --> S[Organize Output Files];
|
||||
end
|
||||
|
||||
subgraph Merged Map Processing
|
||||
A --> B;
|
||||
B --> C;
|
||||
C --> D;
|
||||
D -- Yes --> E;
|
||||
E --> G;
|
||||
C --> H;
|
||||
H --> I;
|
||||
I --> J;
|
||||
J --> K;
|
||||
K --> S;
|
||||
end
|
||||
|
||||
L --> M;
|
||||
M --> S;
|
||||
```
|
||||
|
||||
This plan ensures that statistics are calculated for the roughness data at the appropriate stage (after loading and transformation, before merging) and correctly included in the metadata file, addressing the issue described in ISSUE-013.
|
||||
@ -1,49 +0,0 @@
|
||||
# Issue: GUI Preview Not Updating on File Drop Without Preset Selection
|
||||
|
||||
**ID:** ISSUE-GUI-PreviewNotUpdatingOnDrop
|
||||
**Date:** 2025-04-28
|
||||
**Status:** Open
|
||||
**Priority:** High
|
||||
|
||||
**Description:**
|
||||
The file preview list (`QTableView` in the main panel) does not populate when files or folders are dropped onto the application if a valid preset has not been explicitly selected *after* the application starts and *before* the drop event.
|
||||
|
||||
**Root Cause:**
|
||||
- The `add_input_paths` method in `gui/main_window.py` correctly calls `update_preview` after files are added via drag-and-drop.
|
||||
- However, the `update_preview` method checks `self.editor_preset_list.currentItem()` to get the selected preset.
|
||||
- If no preset is selected, or if the placeholder "--- Select a Preset ---" is selected, `update_preview` correctly identifies this and returns *before* starting the `PredictionHandler` thread.
|
||||
- This prevents the file prediction from running and the preview table from being populated, even if assets have been added.
|
||||
|
||||
**Expected Behavior:**
|
||||
Dropping files should trigger a preview update based on the last valid preset selected by the user, or potentially prompt the user to select a preset if none has ever been selected. The preview should not remain empty simply because the user hasn't re-clicked the preset list immediately before dropping files.
|
||||
|
||||
**Proposed Solution:**
|
||||
1. Modify `MainWindow` to store the last validly selected preset path.
|
||||
2. Update the `update_preview` method:
|
||||
- Instead of relying solely on `currentItem()`, use the stored last valid preset path if `currentItem()` is invalid or the placeholder.
|
||||
- If no valid preset has ever been selected, display a clear message in the status bar or a dialog box prompting the user to select a preset first.
|
||||
- Ensure the prediction handler is triggered with the correct preset name based on this updated logic.
|
||||
|
||||
**Affected Files:**
|
||||
- `gui/main_window.py`
|
||||
|
||||
**Debugging Steps Taken:**
|
||||
- Confirmed `AssetProcessor.__init__` error was fixed.
|
||||
- Added logging to `PredictionHandler` and `MainWindow`.
|
||||
- Logs confirmed `update_preview` is called on drop, but exits early due to no valid preset being selected via `currentItem()`.
|
||||
## Related Changes (Session 2025-04-28)
|
||||
|
||||
During a recent session, work was done to implement a feature allowing GUI editing of the file preview list, specifically the file status and predicted output name.
|
||||
|
||||
This involved defining a data interface (`ProjectNotes/Data_Structures/Preview_Edit_Interface.md`) and modifying several GUI and backend components.
|
||||
|
||||
Key files modified for this feature include:
|
||||
- `gui/prediction_handler.py`
|
||||
- `gui/preview_table_model.py`
|
||||
- `gui/main_window.py`
|
||||
- `gui/processing_handler.py`
|
||||
- Core `asset_processor` logic
|
||||
|
||||
The changes involved passing the editable `file_list` and `asset_properties` data structure through the prediction handler, to the preview table model, and finally to the processing handler. The preview table model was made editable to allow user interaction. The backend logic was updated to utilize this editable data.
|
||||
|
||||
Debugging steps taken during this implementation included fixing indentation errors and resolving an `AssetProcessor` instantiation issue that occurred during the prediction phase.
|
||||
@ -1,91 +0,0 @@
|
||||
---
|
||||
ID: REFACTOR-001
|
||||
Type: Refactor
|
||||
Status: Resolved
|
||||
Priority: Medium
|
||||
Labels: [refactor, core, image-processing, quality]
|
||||
Created: 2025-04-22
|
||||
Updated: 2025-04-22 # Keep original update date, or use today's? Using today's for now.
|
||||
Related: #ISSUE-010 #asset_processor.py #config.py
|
||||
---
|
||||
|
||||
# [REFACTOR-001]: Refactor Map Merging to Use Source Files Directly
|
||||
|
||||
## Description
|
||||
|
||||
Currently, the `_merge_maps` function in `asset_processor.py` loads map data from temporary files that have already been processed by `_process_maps` (resized, format converted, etc.). This intermediate save/load step can introduce quality degradation, especially if the intermediate files are saved using lossy compression (e.g., JPG) or if bit depth conversions occur before merging. This is particularly noticeable in merged maps like NRMRGH where subtle details from the source Normal map might be lost or altered due to recompression.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Improve Quality:** Modify the map merging process to load channel data directly from the *selected original source files* (after classification and 16-bit prioritization) instead of intermediate processed files, thus avoiding potential quality loss from recompression.
|
||||
2. **Maintain Modularity:** Refactor the processing logic to avoid significant code duplication between individual map processing and the merging process.
|
||||
3. **Preserve Functionality:** Ensure all existing functionality (resizing, gloss inversion, format/bit depth rules, merging logic) is retained and applied correctly in the new structure.
|
||||
|
||||
## Proposed Solution: Restructure with Helper Functions
|
||||
|
||||
Introduce two helper functions within the `AssetProcessor` class:
|
||||
|
||||
1. **`_load_and_transform_source(source_path_rel, map_type, target_resolution_key)`:**
|
||||
* Responsible for loading the specified source file.
|
||||
* Performs initial preparation: BGR->RGB conversion, Gloss->Roughness inversion (if applicable), MASK extraction (if applicable).
|
||||
* Resizes the prepared data to the target resolution.
|
||||
* Returns the resized NumPy array and original source dtype.
|
||||
|
||||
2. **`_save_image(image_data, map_type, resolution_key, asset_base_name, source_info, output_bit_depth_rule, temp_dir)`:**
|
||||
* Encapsulates all logic for saving an image.
|
||||
* Determines final output format and bit depth based on rules and source info.
|
||||
* Performs final data type conversions (e.g., to uint8, uint16, float16).
|
||||
* Performs final color space conversion (RGB->BGR for non-EXR).
|
||||
* Constructs the output filename.
|
||||
* Saves the image using `cv2.imwrite`, including fallback logic (e.g., EXR->PNG).
|
||||
* Returns details of the saved temporary file.
|
||||
|
||||
**Modified Workflow:**
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Input Files] --> B(_inventory_and_classify_files);
|
||||
B --> C{Selected Source Maps Info};
|
||||
|
||||
subgraph Core Processing Logic
|
||||
C --> PIM(_process_individual_map);
|
||||
C --> MFS(_merge_maps_from_source);
|
||||
PIM --> LTS([_load_and_transform_source]);
|
||||
MFS --> LTS;
|
||||
end
|
||||
|
||||
LTS --> ImgData{Loaded, Prepared, Resized Image Data};
|
||||
|
||||
subgraph Saving Logic
|
||||
PIM --> SI([_save_image]);
|
||||
MFS --> SI;
|
||||
ImgData --> SI; // Pass image data to save helper
|
||||
end
|
||||
|
||||
SI --> SaveResults{Saved Temp File Details};
|
||||
|
||||
SaveResults --> Results(Processing Results); // Collect results from saving
|
||||
|
||||
Results --> Meta(_generate_metadata_file);
|
||||
Meta --> Org(_organize_output_files);
|
||||
Org --> Final[Final Output Structure];
|
||||
```
|
||||
|
||||
**Function Changes:**
|
||||
|
||||
* **`_process_maps`** was renamed to `_process_individual_map`, responsible only for maps *not* used in merges. It calls `_load_and_transform_source` and `_save_image`.
|
||||
* **`_merge_maps`** was replaced by `_merge_maps_from_source`. It identifies required source paths, calls `_load_and_transform_source` for each input at each target resolution, merges the results, determines saving parameters, and calls `_save_image`.
|
||||
* The main `process` loop coordinates calls to the new functions and handles caching.
|
||||
|
||||
## Resolution (2025-04-22)
|
||||
|
||||
The refactoring described above was implemented in `asset_processor.py`. The `_merge_maps_from_source` function now utilizes the `_load_and_transform_source` and `_save_image` helpers, loading data directly from the classified source files instead of intermediate processed files. User testing confirmed the changes work correctly and improve merged map quality.
|
||||
|
||||
## Acceptance Criteria (Optional)
|
||||
|
||||
* [X] Merged maps (e.g., NRMRGH) are generated correctly using data loaded directly from selected source files.
|
||||
* [X] Visual inspection confirms improved quality/reduced artifacts in merged maps compared to the previous method.
|
||||
* [X] Individual maps (not part of any merge rule) are still processed and saved correctly.
|
||||
* [X] All existing configuration options (resolutions, bit depth rules, format rules, gloss inversion) function as expected within the new structure.
|
||||
* [X] Processing time remains within acceptable limits (potential performance impact of repeated source loading/resizing needs monitoring).
|
||||
* [X] Code remains modular and maintainable, with minimal duplication of core logic.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user