Rusfort/Asset-Frameworker
1
0
Fork 0
Code Issues 24 Pull Requests Actions Packages Projects 3 Releases Wiki Activity

Compare commits

base: Rusfort:1c1620d91a02c9ffbdaf9c4526a9950aea3afc59
Branches Tags
Rusfort:Dev
Rusfort:GUI-and-Configs
Rusfort:Stable
Rusfort:Pipeline
..
compare: Rusfort:8ae9eaed3582175899de70e82b0bedeb3099e3fd
Branches Tags
Rusfort:Dev
Rusfort:GUI-and-Configs
Rusfort:Stable
Rusfort:Pipeline

No commits in common. "1c1620d91a02c9ffbdaf9c4526a9950aea3afc59" and "8ae9eaed3582175899de70e82b0bedeb3099e3fd" have entirely different histories.
1c1620d91a ... 8ae9eaed35

8 changed files with 557 additions and 1050 deletions
Show Stats Expand all files Collapse all files

167
compilation_assessment_plan.md
View File

@ -1,167 +0,0 @@
# Plan: Assessing Compilation of Asset Processor with PyInstaller and Cython
## Objective
To assess the feasibility and create a plan for compiling the Asset Processor project into standalone executables using PyInstaller, incorporating Cython for general speedup and source code obfuscation. A key requirement is to maintain user access to, and the ability to modify, configuration files (like `user_settings.json`, `asset_type_definitions.json`, etc.) and `Preset` files post-compilation.
---
## Phase 1: Initial Analysis & Information Gathering
* **Project Dependencies (from [`requirements.txt`](requirements.txt:1)):**
* `opencv-python`
* `numpy`
* `openexr`
* `PySide6`
* `py7zr`
* `rarfile`
* `requests`
* *Note: `PySide6`, `opencv-python`, and `openexr` may require special handling with PyInstaller (e.g., hidden imports, hooks).*
* **Configuration Loading (based on [`configuration.py`](configuration.py:1)):**
* Configuration files (`app_settings.json`, `llm_settings.json`, `asset_type_definitions.json`, `file_type_definitions.json`, `user_settings.json`, `suppliers.json`) are loaded from a `config/` subdirectory relative to [`configuration.py`](configuration.py:1).
* Preset files are loaded from a `Presets/` subdirectory relative to [`configuration.py`](configuration.py:1).
* `BASE_DIR` is `Path(__file__).parent`, which will refer to the bundled location in a PyInstaller build.
* [`user_settings.json`](configuration.py:16) is designed for overrides and is a candidate for external management.
* Saving functions write back to these relative paths, which needs adaptation.
* **Potential Cython Candidates:**
* Modules within the `processing/` directory.
* Specifically: `processing/utils/image_processing_utils.py` and individual stage files in `processing/pipeline/stages/` (e.g., `alpha_extraction_to_mask.py`, `gloss_to_rough_conversion.py`, etc.).
* Other modules (e.g., `processing/pipeline/orchestrator.py`) could be Cythonized primarily for obfuscation.
* **User-Accessible Files (Defaults):**
* The `config/` directory (containing `app_settings.json`, `asset_type_definitions.json`, `file_type_definitions.json`, `llm_settings.json`, `suppliers.json`).
* The `Presets/` directory and its contents.
---
## Phase 2: Strategy Development
1. **Cython Strategy:**
* **Build Integration:** Utilize a `setup.py` script with `setuptools` and `Cython.Build.cythonize` to compile `.py` files into C extensions (`.pyd` on Windows, `.so` on Linux/macOS).
* **Candidate Prioritization:** Focus on `processing/` modules for performance gains and obfuscation.
* **Compatibility & Challenges:**
* GUI modules (PySide6) are generally left as Python.
* Ensure compatibility with OpenCV, NumPy, and OpenEXR.
* Address potential issues with highly dynamic Python code.
* Consider iterative conversion to `.pyx` files with C-style type annotations for maximum performance in identified hot spots.
* **Obfuscation:** The primary goal for many modules might be obfuscation rather than pure speedup.
2. **PyInstaller Strategy:**
* **Bundle Type:** One-directory bundle (`--onedir`) is recommended for easier debugging and data file management.
* **Data Files (`.spec` file `datas` section):**
* Bundle default `config/` directory (containing `app_settings.json`, `asset_type_definitions.json`, `file_type_definitions.json`, `llm_settings.json`, `suppliers.json`).
* Bundle default `Presets/` directory.
* Include any other necessary GUI assets (icons, etc.).
* Consider bundling the `blender_addon/` if it's to be deployed with the app.
* **Hidden Imports & Hooks (`.spec` file):**
* Add explicit `hiddenimports` for `PySide6`, `opencv-python`, `openexr`, and any other problematic libraries.
* Utilize or create PyInstaller hooks if necessary.
* **Console Window:** Disable for GUI application (`console=False`).
3. **User-Accessible Files & First-Time Setup Strategy:**
* **First-Run Detection:** Application checks for a marker file or stored configuration path.
* **First-Time Setup UI (PySide6 Dialog):**
* **Configuration Location Choice:**
* Option A (Recommended): Store in user profile (e.g., `Documents/AssetProcessor` or `AppData/Roaming/AssetProcessor`).
* Option B (Advanced): User chooses a custom folder.
* The application copies default `config/` (excluding `app_settings.json` but including other definition files) and `Presets/` to the chosen location.
* The chosen path is saved.
* **Key Application Settings Configuration (saved to `user_settings.json` in user's chosen location):**
* Default Library Output Path (`OUTPUT_BASE_DIR`).
* Asset Structure (`OUTPUT_DIRECTORY_PATTERN`).
* Image Output Formats (`OUTPUT_FORMAT_16BIT_PRIMARY`, `OUTPUT_FORMAT_16BIT_FALLBACK`, `OUTPUT_FORMAT_8BIT`).
* JPG Threshold (`RESOLUTION_THRESHOLD_FOR_JPG`).
* Blender Paths (`DEFAULT_NODEGROUP_BLEND_PATH`, `DEFAULT_MATERIALS_BLEND_PATH`, `BLENDER_EXECUTABLE_PATH`).
* **Configuration Loading Logic Modification ([`configuration.py`](configuration.py:1)):**
* `BASE_DIR` for user-modifiable files will point to the user-chosen location.
* `app_settings.json` (master defaults) always loaded from the bundle.
* `user_settings.json` loaded from the user-chosen location, containing overrides.
* Other definition files and `Presets` loaded from the user-chosen location, with a fallback/re-copy mechanism from bundled defaults if missing.
* **Saving Logic Modification ([`configuration.py`](configuration.py:1)):**
* All configuration saving functions will write to the user-chosen configuration location. Bundled defaults remain read-only post-installation.
---
## Phase 3: Outline of Combined Build Process
1. **Environment Setup (Developer):** Install Python, Cython, PyInstaller, and project dependencies.
2. **Cythonization (`setup.py`):**
* Create `setup.py` using `setuptools` and `Cython.Build.cythonize`.
* List `.py` files/modules for compilation (e.g., `processing.utils.image_processing_utils`, `processing.pipeline.stages.*`).
* Include `numpy.get_include()` if Cython files use NumPy C-API.
* Run `python setup.py build_ext --inplace` to generate `.pyd`/`.so` files.
3. **PyInstaller Packaging (`.spec` file):**
* Generate initial `AssetProcessor.spec` with `pyinstaller --name AssetProcessor main.py`.
* Modify `.spec` file:
* `datas`: Add default `config/` and `Presets/` directories, and other assets.
* `hiddenimports`: List modules for `PySide6`, `opencv-python`, etc.
* `excludes`: Optionally exclude original `.py` files for Cythonized modules.
* Set `onedir = True`, `onefile = False`, `console = False`.
* Run `pyinstaller AssetProcessor.spec` to create `dist/AssetProcessor`.
4. **Post-Build Steps (Optional):**
* Clean up original `.py` files from `dist/` if obfuscation is paramount.
* Archive `dist/AssetProcessor` for distribution (ZIP, installer).
---
## Phase 4: Distribution Structure
**Inside `dist/AssetProcessor/` (Distribution Package):**
* `AssetProcessor.exe` (or platform equivalent)
* Core Python and library dependencies (DLLs/SOs)
* Cythonized modules (`.pyd`/`.so` files, e.g., `processing/utils/image_processing_utils.pyd`)
* Non-Cythonized Python modules (`.pyc` files)
* Bundled default `config/` directory (with `app_settings.json`, `asset_type_definitions.json`, etc.)
* Bundled default `Presets/` directory (with `_template.json`, `Dinesen.json`, etc.)
* Other GUI assets (icons, etc.)
* Potentially `blender_addon/` files if bundled.
**User's Configuration Directory (e.g., `Documents/AssetProcessor/`, created on first run):**
* `user_settings.json` (user's choices for paths, formats, etc.)
* Copied `config/` directory (for user modification of `asset_type_definitions.json`, etc.)
* Copied `Presets/` directory (for user modification/additions)
* Marker file for first-time setup choice.
---
## Phase 5: Plan for Testing & Validation
1. **Core Functionality:** Test GUI operations, Directory Monitor, CLI (if applicable).
2. **Configuration System:**
* Verify first-time setup UI, config location choice, copying of defaults.
* Confirm loading from and saving to the user's chosen config location.
* Test modification of user configs and application's reflection of changes.
3. **Dependency Checks:** Ensure bundled libraries (PySide6, OpenCV) function correctly.
4. **Performance (Cython):** Basic comparison of critical operations (Python vs. Cythonized).
5. **Obfuscation (Cython):** Verify absence of original `.py` files for Cythonized modules in distribution (if desired) and that `.pyd`/`.so` files are used.
6. **Cross-Platform Testing:** Repeat build and test process on all target OS.
---
## Phase 6: Documentation Outline
1. **Developer/Build Documentation:**
* Build environment setup.
* `setup.py` (Cython) and `pyinstaller` command usage.
* Structure of `setup.py` and `.spec` file, key configurations.
* Troubleshooting common build issues.
2. **User Documentation:**
* First-time setup guide (config location, initial settings).
* Managing user-specific configurations and presets (location, backup).
* How to reset to default configurations.
---
## Phase 7: Risk Assessment & Mitigation (Brief)
* **Risk:** Cython compilation issues.
* **Mitigation:** Incremental compilation, selective Cythonization.
* **Risk:** PyInstaller packaging complexities.
* **Mitigation:** Thorough testing, community hooks, iterative `.spec` refinement.
* **Risk:** Logic errors in new configuration loading/saving.
* **Mitigation:** Careful coding, detailed testing of config pathways.
* **Risk:** Cython performance not meeting expectations.
* **Mitigation:** Profile Python code first; focus Cython on CPU-bound loops.
* **Risk:** Increased build complexity.
* **Mitigation:** Automate build steps with scripts.

780
configuration.py
View File

@ -1,42 +1,26 @@
import json
import os
import sys
import shutil
from pathlib import Path
import logging
import re
import collections.abc
from typing import Optional, Union
from typing import Optional
log = logging.getLogger(__name__)
# This BASE_DIR is primarily for fallback when not bundled or for locating bundled resources relative to the script.
_SCRIPT_DIR = Path(__file__).resolve().parent
BASE_DIR = Path(__file__).parent
APP_SETTINGS_PATH = BASE_DIR / "config" / "app_settings.json"
LLM_SETTINGS_PATH = BASE_DIR / "config" / "llm_settings.json"
ASSET_TYPE_DEFINITIONS_PATH = BASE_DIR / "config" / "asset_type_definitions.json"
FILE_TYPE_DEFINITIONS_PATH = BASE_DIR / "config" / "file_type_definitions.json"
USER_SETTINGS_PATH = BASE_DIR / "config" / "user_settings.json"
SUPPLIERS_CONFIG_PATH = BASE_DIR / "config" / "suppliers.json"
PRESETS_DIR = BASE_DIR / "Presets"
class ConfigurationError(Exception):
"""Custom exception for configuration loading errors."""
pass
def _get_user_config_path_placeholder() -> Optional[Path]:
"""
Placeholder function. In a real scenario, this would retrieve the
saved user configuration path (e.g., from a settings file).
Returns None if not set, triggering first-time setup behavior.
"""
# For this subtask, we assume this path is determined externally and passed to Configuration.
# If we were to implement the settings.ini check here, it would look like:
# try:
# app_data_dir = Path(os.getenv('APPDATA')) / "AssetProcessor"
# settings_ini = app_data_dir / "settings.ini"
# if settings_ini.exists():
# with open(settings_ini, 'r') as f:
# path_str = f.read().strip()
# return Path(path_str)
# except Exception:
# return None
return None
def _get_base_map_type(target_map_string: str) -> str:
"""Extracts the base map type (e.g., 'COL') from a potentially numbered string ('COL-1')."""
# Use regex to find the leading alphabetical part
@ -108,252 +92,53 @@ def _deep_merge_dicts(base_dict: dict, override_dict: dict) -> dict:
class Configuration:
"""
Loads and provides access to core settings combined with a specific preset,
managing bundled and user-specific configuration paths.
Loads and provides access to core settings combined with a specific preset.
"""
BASE_DIR_APP_BUNDLED_CONFIG_SUBDIR_NAME = "config"
PRESETS_DIR_APP_BUNDLED_NAME = "Presets"
USER_SETTINGS_FILENAME = "user_settings.json"
APP_SETTINGS_FILENAME = "app_settings.json"
ASSET_TYPE_DEFINITIONS_FILENAME = "asset_type_definitions.json"
FILE_TYPE_DEFINITIONS_FILENAME = "file_type_definitions.json"
LLM_SETTINGS_FILENAME = "llm_settings.json"
SUPPLIERS_CONFIG_FILENAME = "suppliers.json"
USER_CONFIG_SUBDIR_NAME = "config" # Subdirectory within user's chosen config root for most jsons
USER_PRESETS_SUBDIR_NAME = "Presets" # Subdirectory within user's chosen config root for presets
def __init__(self, preset_name: str, base_dir_user_config: Optional[Path] = None, is_first_run_setup: bool = False):
def __init__(self, preset_name: str):
"""
Loads core config, user overrides, and the specified preset file.
Args:
preset_name: The name of the preset (without .json extension).
base_dir_user_config: The root path for user-specific configurations.
If None, loading of user-specific files will be skipped or may fail.
is_first_run_setup: Flag indicating if this is part of the initial setup
process where user config dir might be empty and fallbacks
should not aggressively try to copy from bundle until UI confirms.
Raises:
ConfigurationError: If critical configurations cannot be loaded/validated.
ConfigurationError: If core config or preset cannot be loaded/validated.
"""
log.debug(f"Initializing Configuration with preset: '{preset_name}', user_config_dir: '{base_dir_user_config}', first_run_flag: {is_first_run_setup}")
self._preset_filename_stem = preset_name
self.base_dir_user_config: Optional[Path] = base_dir_user_config
self.is_first_run_setup = is_first_run_setup
self.base_dir_app_bundled: Path = self._determine_base_dir_app_bundled()
log.debug(f"Initializing Configuration with preset filename stem: '{preset_name}'")
self._preset_filename_stem = preset_name # Store the stem used for loading
log.info(f"Determined BASE_DIR_APP_BUNDLED: {self.base_dir_app_bundled}")
log.info(f"Using BASE_DIR_USER_CONFIG: {self.base_dir_user_config}")
# 1. Load core settings
self._core_settings: dict = self._load_core_config()
# 1. Load core application settings (always from bundled)
app_settings_path = self.base_dir_app_bundled / self.BASE_DIR_APP_BUNDLED_CONFIG_SUBDIR_NAME / self.APP_SETTINGS_FILENAME
self._core_settings: dict = self._load_json_file(
app_settings_path,
is_critical=True,
description="Core application settings"
)
# 2. Load asset type definitions
self._asset_type_definitions: dict = self._load_asset_type_definitions()
# 2. Load user settings (from user config dir, if provided)
user_settings_overrides: dict = {}
if self.base_dir_user_config:
user_settings_file_path = self.base_dir_user_config / self.USER_SETTINGS_FILENAME
user_settings_overrides = self._load_json_file(
user_settings_file_path,
is_critical=False, # Not critical if missing, especially on first run
description=f"User settings from {user_settings_file_path}"
) or {} # Ensure it's a dict
else:
log.info(f"{self.USER_SETTINGS_FILENAME} not loaded: User config directory not set.")
# 3. Load file type definitions
self._file_type_definitions: dict = self._load_file_type_definitions()
# 3. Deep merge user settings onto core settings
# 4. Load user settings
user_settings_overrides: dict = self._load_user_settings()
# 5. Deep merge user settings onto core settings
if user_settings_overrides:
log.info(f"Applying user setting overrides to core settings.")
log.info("Applying user setting overrides to core settings.")
# _deep_merge_dicts modifies self._core_settings in place
_deep_merge_dicts(self._core_settings, user_settings_overrides)
# 4. Load other definition files (from user config dir, with fallback from bundled)
self._asset_type_definitions: dict = self._load_definition_file_with_fallback(
self.ASSET_TYPE_DEFINITIONS_FILENAME, "ASSET_TYPE_DEFINITIONS"
)
self._file_type_definitions: dict = self._load_definition_file_with_fallback(
self.FILE_TYPE_DEFINITIONS_FILENAME, "FILE_TYPE_DEFINITIONS"
)
self._llm_settings: dict = self._load_definition_file_with_fallback(
self.LLM_SETTINGS_FILENAME, None # LLM settings might be flat (no root key)
)
self._suppliers_config: dict = self._load_definition_file_with_fallback(
self.SUPPLIERS_CONFIG_FILENAME, None # Suppliers config is flat
)
# 6. Load LLM settings
self._llm_settings: dict = self._load_llm_config()
# 5. Load preset settings (from user config dir, with fallback from bundled)
self._preset_settings: dict = self._load_preset_with_fallback(self._preset_filename_stem)
# 7. Load preset settings (conceptually overrides combined base + user for shared keys)
self._preset_settings: dict = self._load_preset(self._preset_filename_stem) # Use the stored stem
# Store the actual preset name read from the file content
self.actual_internal_preset_name = self._preset_settings.get("preset_name", self._preset_filename_stem)
log.info(f"Configuration instance: Loaded preset file '{self._preset_filename_stem}.json', internal preset_name is '{self.actual_internal_preset_name}'")
# 6. Validate and compile (after all base/user/preset settings are established)
# 8. Validate and compile (after all base/user/preset settings are established)
self._validate_configs()
self._compile_regex_patterns()
log.info(f"Configuration loaded successfully using preset: '{self.actual_internal_preset_name}'")
def _determine_base_dir_app_bundled(self) -> Path:
"""Determines the base directory for bundled application resources."""
if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
# Running in a PyInstaller bundle
log.debug(f"Running as bundled app, _MEIPASS: {sys._MEIPASS}")
return Path(sys._MEIPASS)
else:
# Running as a script
log.debug(f"Running as script, using _SCRIPT_DIR: {_SCRIPT_DIR}")
return _SCRIPT_DIR
def _ensure_dir_exists(self, dir_path: Path):
"""Ensures a directory exists, creating it if necessary."""
try:
if not dir_path.exists():
log.info(f"Directory not found, creating: {dir_path}")
dir_path.mkdir(parents=True, exist_ok=True)
elif not dir_path.is_dir():
raise ConfigurationError(f"Expected directory but found file: {dir_path}")
except OSError as e:
raise ConfigurationError(f"Failed to create or access directory {dir_path}: {e}")
def _copy_default_if_missing(self, user_target_path: Path, bundled_source_subdir: str, filename: str) -> bool:
"""
Copies a default file from the bundled location to the user config directory
if it's missing in the user directory. This is for post-first-time-setup fallback.
"""
if not self.base_dir_user_config:
log.error(f"Cannot copy default for '{filename}': base_dir_user_config is not set.")
return False
if user_target_path.exists():
log.debug(f"User file '{user_target_path}' already exists. No copy needed from bundle.")
return False
# This fallback copy should NOT happen during the initial UI-driven setup phase
# where the UI is responsible for the first population of the user directory.
# It's for subsequent runs where a user might have deleted a file.
if self.is_first_run_setup:
log.debug(f"'{filename}' missing in user dir during first_run_setup phase. UI should handle initial copy. Skipping fallback copy.")
return False # File is missing, but UI should handle it.
bundled_file_path = self.base_dir_app_bundled / bundled_source_subdir / filename
if not bundled_file_path.is_file():
log.warning(f"Default bundled file '{bundled_file_path}' not found. Cannot copy to user location '{user_target_path}'.")
return False
log.warning(f"User file '{user_target_path}' is missing. Attempting to restore from bundled default: '{bundled_file_path}'.")
try:
self._ensure_dir_exists(user_target_path.parent)
shutil.copy2(bundled_file_path, user_target_path)
log.info(f"Successfully copied '{bundled_file_path}' to '{user_target_path}'.")
return True # File was copied
except Exception as e:
log.error(f"Failed to copy '{bundled_file_path}' to '{user_target_path}': {e}")
return False # Copy failed
def _load_json_file(self, file_path: Optional[Path], is_critical: bool = False, description: str = "configuration") -> dict:
"""Loads a JSON file, handling errors. Returns empty dict if not found and not critical."""
if not file_path:
if is_critical:
raise ConfigurationError(f"Critical {description} file path is not defined.")
log.debug(f"{description} file path is not defined. Returning empty dict.")
return {}
log.debug(f"Attempting to load {description} from: {file_path}")
if not file_path.is_file():
if is_critical:
raise ConfigurationError(f"Critical {description} file not found: {file_path}")
log.info(f"{description} file not found: {file_path}. Returning empty dict.")
return {}
try:
with open(file_path, 'r', encoding='utf-8') as f:
settings = json.load(f)
log.debug(f"{description} loaded successfully from {file_path}.")
return settings
except json.JSONDecodeError as e:
msg = f"Failed to parse {description} file {file_path}: Invalid JSON - {e}"
if is_critical: raise ConfigurationError(msg)
log.warning(msg + ". Returning empty dict.")
return {}
except Exception as e:
msg = f"Failed to read {description} file {file_path}: {e}"
if is_critical: raise ConfigurationError(msg)
log.warning(msg + ". Returning empty dict.")
return {}
def _load_definition_file_with_fallback(self, filename: str, root_key: Optional[str] = None) -> dict:
"""
Loads a definition JSON file from the user config subdir.
If not found and not first_run_setup, attempts to copy from bundled config subdir and then loads it.
If base_dir_user_config is not set, loads directly from bundled (read-only).
"""
data = {}
user_file_path = None
if self.base_dir_user_config:
user_file_path = self.base_dir_user_config / self.USER_CONFIG_SUBDIR_NAME / filename
data = self._load_json_file(user_file_path, is_critical=False, description=f"User {filename}")
if not data: # If not found or failed to load from user path
# Attempt fallback copy only if not in the initial setup phase by UI
# and if the file was genuinely missing (not a parse error for an existing file)
if not user_file_path.exists() and not self.is_first_run_setup:
if self._copy_default_if_missing(user_file_path, self.BASE_DIR_APP_BUNDLED_CONFIG_SUBDIR_NAME, filename):
data = self._load_json_file(user_file_path, is_critical=False, description=f"User {filename} after copy")
else:
# No user_config_dir, load directly from bundled (read-only)
log.warning(f"User config directory not set. Loading '{filename}' from bundled defaults (read-only).")
bundled_path = self.base_dir_app_bundled / self.BASE_DIR_APP_BUNDLED_CONFIG_SUBDIR_NAME / filename
data = self._load_json_file(bundled_path, is_critical=False, description=f"Bundled {filename}")
if not data:
# If still no data, it's an issue, especially for critical definitions
is_critical_def = filename in [self.ASSET_TYPE_DEFINITIONS_FILENAME, self.FILE_TYPE_DEFINITIONS_FILENAME]
err_msg = f"Failed to load '{filename}' from user dir '{user_file_path if user_file_path else 'N/A'}' or bundled defaults. Critical functionality may be affected."
if is_critical_def: raise ConfigurationError(err_msg)
log.error(err_msg)
return {}
if root_key:
if root_key not in data:
raise ConfigurationError(f"Key '{root_key}' not found in loaded {filename} data: {data.keys()}")
content = data[root_key]
# Ensure content is a dictionary if a root_key is expected to yield one
if not isinstance(content, dict):
raise ConfigurationError(f"Content under root key '{root_key}' in {filename} must be a dictionary, got {type(content)}.")
return content
return data # For flat files
def _load_preset_with_fallback(self, preset_name_stem: str) -> dict:
"""
Loads a preset JSON file from the user's Presets subdir.
If not found and not first_run_setup, attempts to copy from bundled Presets and then loads it.
If base_dir_user_config is not set, loads directly from bundled (read-only).
"""
preset_filename = f"{preset_name_stem}.json"
preset_data = {}
user_preset_file_path = None
if self.base_dir_user_config:
user_presets_dir = self.base_dir_user_config / self.USER_PRESETS_SUBDIR_NAME
user_preset_file_path = user_presets_dir / preset_filename
preset_data = self._load_json_file(user_preset_file_path, is_critical=False, description=f"User preset '{preset_filename}'")
if not preset_data: # If not found or failed to load
if not user_preset_file_path.exists() and not self.is_first_run_setup:
if self._copy_default_if_missing(user_preset_file_path, self.PRESETS_DIR_APP_BUNDLED_NAME, preset_filename):
preset_data = self._load_json_file(user_preset_file_path, is_critical=False, description=f"User preset '{preset_filename}' after copy")
else:
log.warning(f"User config directory not set. Loading preset '{preset_filename}' from bundled defaults (read-only).")
bundled_presets_dir = self.base_dir_app_bundled / self.PRESETS_DIR_APP_BUNDLED_NAME
bundled_preset_file_path = bundled_presets_dir / preset_filename
# Presets are generally critical for operation if one is specified
preset_data = self._load_json_file(bundled_preset_file_path, is_critical=True, description=f"Bundled preset '{preset_filename}'")
if not preset_data:
raise ConfigurationError(f"Preset file '{preset_filename}' could not be loaded from user dir '{user_preset_file_path if user_preset_file_path else 'N/A'}' or bundled defaults.")
return preset_data
log.info(f"Configuration loaded successfully using preset: '{self.actual_internal_preset_name}'") # Changed self.preset_name to self.actual_internal_preset_name
def _compile_regex_patterns(self):
@ -452,6 +237,118 @@ class Configuration:
log.debug("Finished compiling regex patterns.")
def _load_core_config(self) -> dict:
"""Loads settings from the core app_settings.json file."""
log.debug(f"Loading core config from: {APP_SETTINGS_PATH}")
if not APP_SETTINGS_PATH.is_file():
raise ConfigurationError(f"Core configuration file not found: {APP_SETTINGS_PATH}")
try:
with open(APP_SETTINGS_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
log.debug(f"Core config loaded successfully.")
return settings
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse core configuration file {APP_SETTINGS_PATH}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read core configuration file {APP_SETTINGS_PATH}: {e}")
def _load_llm_config(self) -> dict:
"""Loads settings from the llm_settings.json file."""
log.debug(f"Loading LLM config from: {LLM_SETTINGS_PATH}")
if not LLM_SETTINGS_PATH.is_file():
# Log a warning but don't raise an error, allow fallback if possible
log.warning(f"LLM configuration file not found: {LLM_SETTINGS_PATH}. LLM features might be disabled or use defaults.")
return {}
try:
with open(LLM_SETTINGS_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
log.debug(f"LLM config loaded successfully.")
return settings
except json.JSONDecodeError as e:
log.error(f"Failed to parse LLM configuration file {LLM_SETTINGS_PATH}: Invalid JSON - {e}")
return {}
except Exception as e:
log.error(f"Failed to read LLM configuration file {LLM_SETTINGS_PATH}: {e}")
return {}
def _load_preset(self, preset_name: str) -> dict:
"""Loads the specified preset JSON file."""
log.debug(f"Loading preset: '{preset_name}' from {PRESETS_DIR}")
if not PRESETS_DIR.is_dir():
raise ConfigurationError(f"Presets directory not found: {PRESETS_DIR}")
preset_file = PRESETS_DIR / f"{preset_name}.json"
if not preset_file.is_file():
raise ConfigurationError(f"Preset file not found: {preset_file}")
try:
with open(preset_file, 'r', encoding='utf-8') as f:
preset_data = json.load(f)
log.debug(f"Preset '{preset_name}' loaded successfully.")
return preset_data
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse preset file {preset_file}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read preset file {preset_file}: {e}")
def _load_asset_type_definitions(self) -> dict:
"""Loads asset type definitions from the asset_type_definitions.json file."""
log.debug(f"Loading asset type definitions from: {ASSET_TYPE_DEFINITIONS_PATH}")
if not ASSET_TYPE_DEFINITIONS_PATH.is_file():
raise ConfigurationError(f"Asset type definitions file not found: {ASSET_TYPE_DEFINITIONS_PATH}")
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "ASSET_TYPE_DEFINITIONS" not in data:
raise ConfigurationError(f"Key 'ASSET_TYPE_DEFINITIONS' not found in {ASSET_TYPE_DEFINITIONS_PATH}")
settings = data["ASSET_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
raise ConfigurationError(f"'ASSET_TYPE_DEFINITIONS' in {ASSET_TYPE_DEFINITIONS_PATH} must be a dictionary.")
log.debug(f"Asset type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}")
def _load_file_type_definitions(self) -> dict:
"""Loads file type definitions from the file_type_definitions.json file."""
log.debug(f"Loading file type definitions from: {FILE_TYPE_DEFINITIONS_PATH}")
if not FILE_TYPE_DEFINITIONS_PATH.is_file():
raise ConfigurationError(f"File type definitions file not found: {FILE_TYPE_DEFINITIONS_PATH}")
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "FILE_TYPE_DEFINITIONS" not in data:
raise ConfigurationError(f"Key 'FILE_TYPE_DEFINITIONS' not found in {FILE_TYPE_DEFINITIONS_PATH}")
settings = data["FILE_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
raise ConfigurationError(f"'FILE_TYPE_DEFINITIONS' in {FILE_TYPE_DEFINITIONS_PATH} must be a dictionary.")
log.debug(f"File type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
raise ConfigurationError(f"Failed to parse file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
except Exception as e:
raise ConfigurationError(f"Failed to read file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}")
def _load_user_settings(self) -> dict:
"""Loads user override settings from config/user_settings.json."""
log.debug(f"Attempting to load user settings from: {USER_SETTINGS_PATH}")
if not USER_SETTINGS_PATH.is_file():
log.info(f"User settings file not found: {USER_SETTINGS_PATH}. Proceeding without user overrides.")
return {}
try:
with open(USER_SETTINGS_PATH, 'r', encoding='utf-8') as f:
settings = json.load(f)
log.info(f"User settings loaded successfully from {USER_SETTINGS_PATH}.")
return settings
except json.JSONDecodeError as e:
log.warning(f"Failed to parse user settings file {USER_SETTINGS_PATH}: Invalid JSON - {e}. Using empty user settings.")
return {}
except Exception as e:
log.warning(f"Failed to read user settings file {USER_SETTINGS_PATH}: {e}. Using empty user settings.")
return {}
def _validate_configs(self):
"""Performs basic validation checks on loaded settings."""
@ -548,14 +445,9 @@ class Configuration:
@property
def supplier_name(self) -> str: # From preset
def supplier_name(self) -> str:
return self._preset_settings.get('supplier_name', 'DefaultSupplier')
@property
def suppliers_config(self) -> dict: # From suppliers.json
"""Returns the loaded suppliers configuration."""
return self._suppliers_config
@property
def internal_display_preset_name(self) -> str:
"""Returns the 'preset_name' field from within the loaded preset JSON,
@ -824,64 +716,9 @@ class Configuration:
return self._core_settings.get('LOW_RESOLUTION_THRESHOLD', 512)
@property
def FILE_TYPE_DEFINITIONS(self) -> dict: # Kept for compatibility if used directly
def FILE_TYPE_DEFINITIONS(self) -> dict:
return self._file_type_definitions
# --- Save Methods ---
def _save_json_to_user_config(self, data_to_save: dict, filename: str, subdir: Optional[str] = None, is_root_key_data: Optional[str] = None):
"""Helper to save a dictionary to a JSON file in the user config directory."""
if not self.base_dir_user_config:
raise ConfigurationError(f"Cannot save {filename}: User config directory (base_dir_user_config) is not set.")
target_dir = self.base_dir_user_config
if subdir:
target_dir = target_dir / subdir
self._ensure_dir_exists(target_dir)
path = target_dir / filename
data_for_json = {is_root_key_data: data_to_save} if is_root_key_data else data_to_save
log.debug(f"Saving data to: {path}")
try:
with open(path, 'w', encoding='utf-8') as f:
json.dump(data_for_json, f, indent=4)
log.info(f"Data saved successfully to {path}")
except Exception as e:
log.error(f"Failed to save file {path}: {e}")
raise ConfigurationError(f"Failed to save {filename}: {e}")
def save_user_settings(self, settings_dict: dict):
"""Saves the provided settings dictionary to user_settings.json in the user config directory."""
self._save_json_to_user_config(settings_dict, self.USER_SETTINGS_FILENAME)
def save_llm_settings(self, settings_dict: dict):
"""Saves LLM settings to the user config directory's 'config' subdir."""
self._save_json_to_user_config(settings_dict, self.LLM_SETTINGS_FILENAME, subdir=self.USER_CONFIG_SUBDIR_NAME)
def save_asset_type_definitions(self, data: dict):
"""Saves asset type definitions to the user config directory's 'config' subdir."""
self._save_json_to_user_config(data, self.ASSET_TYPE_DEFINITIONS_FILENAME, subdir=self.USER_CONFIG_SUBDIR_NAME, is_root_key_data="ASSET_TYPE_DEFINITIONS")
def save_file_type_definitions(self, data: dict):
"""Saves file type definitions to the user config directory's 'config' subdir."""
self._save_json_to_user_config(data, self.FILE_TYPE_DEFINITIONS_FILENAME, subdir=self.USER_CONFIG_SUBDIR_NAME, is_root_key_data="FILE_TYPE_DEFINITIONS")
def save_supplier_settings(self, data: dict):
"""Saves supplier settings to the user config directory's 'config' subdir."""
self._save_json_to_user_config(data, self.SUPPLIERS_CONFIG_FILENAME, subdir=self.USER_CONFIG_SUBDIR_NAME)
def save_preset(self, preset_data: dict, preset_name_stem: str):
"""Saves a preset to the user config directory's 'Presets' subdir."""
if not preset_name_stem:
raise ConfigurationError("Preset name stem cannot be empty for saving.")
preset_filename = f"{preset_name_stem}.json"
# Ensure the preset_data itself contains the correct 'preset_name' field
# or update it before saving if necessary.
# For example: preset_data['preset_name'] = preset_name_stem
self._save_json_to_user_config(preset_data, preset_filename, subdir=self.USER_PRESETS_SUBDIR_NAME)
@property
def keybind_config(self) -> dict[str, list[str]]:
"""
@ -905,60 +742,275 @@ class Configuration:
# For now, we rely on the order they appear in the config.
return keybinds
# The global load_base_config() is effectively replaced by Configuration.__init__
# Global save/load functions for individual files are refactored to be methods
# of the Configuration class or called by them, using instance paths.
def load_base_config() -> dict:
"""
Loads base configuration by merging app_settings.json, user_settings.json (if exists),
asset_type_definitions.json, and file_type_definitions.json.
Does not load presets or perform full validation beyond basic file loading.
Returns a dictionary containing the merged settings. If app_settings.json
fails to load, an empty dictionary is returned. If other files
fail, errors are logged, and the function proceeds with what has been loaded.
"""
base_settings = {}
# For example, to get a list of preset names, one might need a static method
# or a function that knows about both bundled and user preset directories.
def get_available_preset_names(base_dir_user_config: Optional[Path], base_dir_app_bundled: Path) -> list[str]:
"""
Gets a list of available preset names (stems) by looking in user presets
and then bundled presets. User presets take precedence.
"""
preset_names = set()
# 1. Load app_settings.json (critical)
if not APP_SETTINGS_PATH.is_file():
log.error(f"Critical: Base application settings file not found: {APP_SETTINGS_PATH}. Returning empty configuration.")
return {}
try:
with open(APP_SETTINGS_PATH, 'r', encoding='utf-8') as f:
base_settings = json.load(f)
log.info(f"Successfully loaded base application settings from: {APP_SETTINGS_PATH}")
except json.JSONDecodeError as e:
log.error(f"Critical: Failed to parse base application settings file {APP_SETTINGS_PATH}: Invalid JSON - {e}. Returning empty configuration.")
return {}
except Exception as e:
log.error(f"Critical: Failed to read base application settings file {APP_SETTINGS_PATH}: {e}. Returning empty configuration.")
return {}
# 2. Attempt to load user_settings.json
user_settings_overrides = {}
if USER_SETTINGS_PATH.is_file():
try:
with open(USER_SETTINGS_PATH, 'r', encoding='utf-8') as f:
user_settings_overrides = json.load(f)
log.info(f"User settings loaded successfully for base_config from {USER_SETTINGS_PATH}.")
except json.JSONDecodeError as e:
log.warning(f"Failed to parse user settings file {USER_SETTINGS_PATH} for base_config: Invalid JSON - {e}. Proceeding without these user overrides.")
except Exception as e:
log.warning(f"Failed to read user settings file {USER_SETTINGS_PATH} for base_config: {e}. Proceeding without these user overrides.")
# Check user presets first
if base_dir_user_config:
user_presets_dir = base_dir_user_config / Configuration.USER_PRESETS_SUBDIR_NAME
if user_presets_dir.is_dir():
for f in user_presets_dir.glob("*.json"):
preset_names.add(f.stem)
# Check bundled presets
bundled_presets_dir = base_dir_app_bundled / Configuration.PRESETS_DIR_APP_BUNDLED_NAME
if bundled_presets_dir.is_dir():
for f in bundled_presets_dir.glob("*.json"):
preset_names.add(f.stem) # Adds if not already present from user dir
# 3. Deep merge user settings onto base_settings
if user_settings_overrides:
log.info("Applying user setting overrides to base_settings in load_base_config.")
# _deep_merge_dicts modifies base_settings in place
_deep_merge_dicts(base_settings, user_settings_overrides)
# 4. Load asset_type_definitions.json (non-critical, merge if successful)
if not ASSET_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"Asset type definitions file not found: {ASSET_TYPE_DEFINITIONS_PATH}. Proceeding without it.")
else:
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
asset_defs_data = json.load(f)
if "ASSET_TYPE_DEFINITIONS" in asset_defs_data:
if isinstance(asset_defs_data["ASSET_TYPE_DEFINITIONS"], dict):
# Merge into base_settings, which might already contain user overrides
base_settings['ASSET_TYPE_DEFINITIONS'] = asset_defs_data["ASSET_TYPE_DEFINITIONS"]
log.info(f"Successfully loaded and merged ASSET_TYPE_DEFINITIONS from: {ASSET_TYPE_DEFINITIONS_PATH}")
else:
log.error(f"Value under 'ASSET_TYPE_DEFINITIONS' in {ASSET_TYPE_DEFINITIONS_PATH} is not a dictionary. Skipping merge.")
else:
log.error(f"Key 'ASSET_TYPE_DEFINITIONS' not found in {ASSET_TYPE_DEFINITIONS_PATH}. Skipping merge.")
except json.JSONDecodeError as e:
log.error(f"Failed to parse asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}. Skipping merge.")
except Exception as e:
log.error(f"Failed to read asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}. Skipping merge.")
# 5. Load file_type_definitions.json (non-critical, merge if successful)
if not FILE_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"File type definitions file not found: {FILE_TYPE_DEFINITIONS_PATH}. Proceeding without it.")
else:
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
file_defs_data = json.load(f)
if "FILE_TYPE_DEFINITIONS" in file_defs_data:
if isinstance(file_defs_data["FILE_TYPE_DEFINITIONS"], dict):
# Merge into base_settings
base_settings['FILE_TYPE_DEFINITIONS'] = file_defs_data["FILE_TYPE_DEFINITIONS"]
log.info(f"Successfully loaded and merged FILE_TYPE_DEFINITIONS from: {FILE_TYPE_DEFINITIONS_PATH}")
else:
log.error(f"Value under 'FILE_TYPE_DEFINITIONS' in {FILE_TYPE_DEFINITIONS_PATH} is not a dictionary. Skipping merge.")
else:
log.error(f"Key 'FILE_TYPE_DEFINITIONS' not found in {FILE_TYPE_DEFINITIONS_PATH}. Skipping merge.")
except json.JSONDecodeError as e:
log.error(f"Failed to parse file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}. Skipping merge.")
except Exception as e:
log.error(f"Failed to read file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}. Skipping merge.")
return base_settings
def save_llm_config(settings_dict: dict):
"""
Saves the provided LLM settings dictionary to llm_settings.json.
"""
log.debug(f"Saving LLM config to: {LLM_SETTINGS_PATH}")
try:
with open(LLM_SETTINGS_PATH, 'w', encoding='utf-8') as f:
json.dump(settings_dict, f, indent=4)
# Use info level for successful save
log.info(f"LLM config saved successfully to {LLM_SETTINGS_PATH}")
except Exception as e:
log.error(f"Failed to save LLM configuration file {LLM_SETTINGS_PATH}: {e}")
# Re-raise as ConfigurationError to signal failure upstream
raise ConfigurationError(f"Failed to save LLM configuration: {e}")
def save_user_config(settings_dict: dict):
"""Saves the provided settings dictionary to user_settings.json."""
log.debug(f"Saving user config to: {USER_SETTINGS_PATH}")
try:
# Ensure parent directory exists (though 'config/' should always exist)
USER_SETTINGS_PATH.parent.mkdir(parents=True, exist_ok=True)
with open(USER_SETTINGS_PATH, 'w', encoding='utf-8') as f:
json.dump(settings_dict, f, indent=4)
log.info(f"User config saved successfully to {USER_SETTINGS_PATH}")
except Exception as e:
log.error(f"Failed to save user configuration file {USER_SETTINGS_PATH}: {e}")
raise ConfigurationError(f"Failed to save user configuration: {e}")
def save_base_config(settings_dict: dict):
"""
Saves the provided settings dictionary to app_settings.json.
"""
log.debug(f"Saving base config to: {APP_SETTINGS_PATH}")
try:
with open(APP_SETTINGS_PATH, 'w', encoding='utf-8') as f:
json.dump(settings_dict, f, indent=4)
log.debug(f"Base config saved successfully.")
except Exception as e:
log.error(f"Failed to save base configuration file {APP_SETTINGS_PATH}: {e}")
raise ConfigurationError(f"Failed to save configuration: {e}")
def load_asset_definitions() -> dict:
"""
Reads config/asset_type_definitions.json.
Returns the dictionary under the "ASSET_TYPE_DEFINITIONS" key.
Handles file not found or JSON errors gracefully (e.g., return empty dict, log error).
"""
log.debug(f"Loading asset type definitions from: {ASSET_TYPE_DEFINITIONS_PATH}")
if not ASSET_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"Asset type definitions file not found: {ASSET_TYPE_DEFINITIONS_PATH}")
return {}
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "ASSET_TYPE_DEFINITIONS" not in data:
log.error(f"Key 'ASSET_TYPE_DEFINITIONS' not found in {ASSET_TYPE_DEFINITIONS_PATH}")
return {}
settings = data["ASSET_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
log.error(f"'ASSET_TYPE_DEFINITIONS' in {ASSET_TYPE_DEFINITIONS_PATH} must be a dictionary.")
return {}
log.debug(f"Asset type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
log.error(f"Failed to parse asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
return {}
except Exception as e:
log.error(f"Failed to read asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}")
return {}
def save_asset_definitions(data: dict):
"""
Takes a dictionary (representing the content for the "ASSET_TYPE_DEFINITIONS" key).
Writes it to config/asset_type_definitions.json under the root key "ASSET_TYPE_DEFINITIONS".
Handles potential I/O errors.
"""
log.debug(f"Saving asset type definitions to: {ASSET_TYPE_DEFINITIONS_PATH}")
try:
with open(ASSET_TYPE_DEFINITIONS_PATH, 'w', encoding='utf-8') as f:
json.dump({"ASSET_TYPE_DEFINITIONS": data}, f, indent=4)
log.info(f"Asset type definitions saved successfully to {ASSET_TYPE_DEFINITIONS_PATH}")
except Exception as e:
log.error(f"Failed to save asset type definitions file {ASSET_TYPE_DEFINITIONS_PATH}: {e}")
raise ConfigurationError(f"Failed to save asset type definitions: {e}")
def load_file_type_definitions() -> dict:
"""
Reads config/file_type_definitions.json.
Returns the dictionary under the "FILE_TYPE_DEFINITIONS" key.
Handles errors gracefully.
"""
log.debug(f"Loading file type definitions from: {FILE_TYPE_DEFINITIONS_PATH}")
if not FILE_TYPE_DEFINITIONS_PATH.is_file():
log.error(f"File type definitions file not found: {FILE_TYPE_DEFINITIONS_PATH}")
return {}
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if "FILE_TYPE_DEFINITIONS" not in data:
log.error(f"Key 'FILE_TYPE_DEFINITIONS' not found in {FILE_TYPE_DEFINITIONS_PATH}")
return {}
settings = data["FILE_TYPE_DEFINITIONS"]
if not isinstance(settings, dict):
log.error(f"'FILE_TYPE_DEFINITIONS' in {FILE_TYPE_DEFINITIONS_PATH} must be a dictionary.")
return {}
log.debug(f"File type definitions loaded successfully.")
return settings
except json.JSONDecodeError as e:
log.error(f"Failed to parse file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: Invalid JSON - {e}")
return {}
except Exception as e:
log.error(f"Failed to read file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}")
return {}
def save_file_type_definitions(data: dict):
"""
Takes a dictionary (representing content for "FILE_TYPE_DEFINITIONS" key).
Writes it to config/file_type_definitions.json under the root key "FILE_TYPE_DEFINITIONS".
Handles errors.
"""
log.debug(f"Saving file type definitions to: {FILE_TYPE_DEFINITIONS_PATH}")
try:
with open(FILE_TYPE_DEFINITIONS_PATH, 'w', encoding='utf-8') as f:
json.dump({"FILE_TYPE_DEFINITIONS": data}, f, indent=4)
log.info(f"File type definitions saved successfully to {FILE_TYPE_DEFINITIONS_PATH}")
except Exception as e:
log.error(f"Failed to save file type definitions file {FILE_TYPE_DEFINITIONS_PATH}: {e}")
raise ConfigurationError(f"Failed to save file type definitions: {e}")
def load_supplier_settings() -> dict:
"""
Reads config/suppliers.json.
Returns the entire dictionary.
Handles file not found (return empty dict) or JSON errors.
If the loaded data is a list (old format), convert it in memory to the new
dictionary format, defaulting normal_map_type to "OpenGL" for each supplier.
"""
log.debug(f"Loading supplier settings from: {SUPPLIERS_CONFIG_PATH}")
if not SUPPLIERS_CONFIG_PATH.is_file():
log.warning(f"Supplier settings file not found: {SUPPLIERS_CONFIG_PATH}. Returning empty dict.")
return {}
try:
with open(SUPPLIERS_CONFIG_PATH, 'r', encoding='utf-8') as f:
data = json.load(f)
if isinstance(data, list):
log.warning(f"Supplier settings in {SUPPLIERS_CONFIG_PATH} is in the old list format. Converting to new dictionary format.")
new_data = {}
for supplier_name in data:
if isinstance(supplier_name, str):
new_data[supplier_name] = {"normal_map_type": "OpenGL"}
else:
log.warning(f"Skipping non-string item '{supplier_name}' during old format conversion of supplier settings.")
log.info(f"Supplier settings converted to new format: {new_data}")
return new_data
if not isinstance(data, dict):
log.error(f"Supplier settings in {SUPPLIERS_CONFIG_PATH} must be a dictionary. Found {type(data)}. Returning empty dict.")
return {}
if not preset_names:
log.warning("No preset files found in user or bundled preset directories.")
# Consider adding a default/template preset if none are found, or ensure one always exists in bundle.
# For now, return empty list.
return sorted(list(preset_names))
log.debug(f"Supplier settings loaded successfully.")
return data
except json.JSONDecodeError as e:
log.error(f"Failed to parse supplier settings file {SUPPLIERS_CONFIG_PATH}: Invalid JSON - {e}. Returning empty dict.")
return {}
except Exception as e:
log.error(f"Failed to read supplier settings file {SUPPLIERS_CONFIG_PATH}: {e}. Returning empty dict.")
return {}
# Global functions like load_asset_definitions, save_asset_definitions etc.
# are now instance methods of the Configuration class (e.g., self.save_asset_type_definitions).
# If any external code was calling these global functions, it will need to be updated
# to instantiate a Configuration object and call its methods, or these global
# functions need to be carefully adapted to instantiate Configuration internally
# or accept a Configuration instance.
# For now, let's assume the primary interaction is via Configuration instance.
# The old global functions below this point are effectively deprecated by the class methods.
# I will remove them to avoid confusion and ensure all save/load operations
# are managed through the Configuration instance with correct path context.
# Removing old global load/save functions as their logic is now
# part of the Configuration class or replaced by its new loading/saving mechanisms.
# load_base_config() - Replaced by Configuration.__init__()
# save_llm_config(settings_dict: dict) - Replaced by Configuration.save_llm_settings()
# save_user_config(settings_dict: dict) - Replaced by Configuration.save_user_settings()
# save_base_config(settings_dict: dict) - Bundled app_settings.json should be read-only.
# load_asset_definitions() -> dict - Replaced by Configuration._load_definition_file_with_fallback() logic
# save_asset_definitions(data: dict) - Replaced by Configuration.save_asset_type_definitions()
# load_file_type_definitions() -> dict - Replaced by Configuration._load_definition_file_with_fallback() logic
# save_file_type_definitions(data: dict) - Replaced by Configuration.save_file_type_definitions()
# load_supplier_settings() -> dict - Replaced by Configuration._load_definition_file_with_fallback() logic
# save_supplier_settings(data: dict) - Replaced by Configuration.save_supplier_settings()
def save_supplier_settings(data: dict):
"""
Takes a dictionary (in the new format).
Writes it directly to config/suppliers.json.
Handles errors.
"""
log.debug(f"Saving supplier settings to: {SUPPLIERS_CONFIG_PATH}")
if not isinstance(data, dict):
log.error(f"Data for save_supplier_settings must be a dictionary. Got {type(data)}.")
raise ConfigurationError(f"Invalid data type for saving supplier settings: {type(data)}")
try:
with open(SUPPLIERS_CONFIG_PATH, 'w', encoding='utf-8') as f:
json.dump(data, f, indent=2) # Using indent=2 as per the example for suppliers.json
log.info(f"Supplier settings saved successfully to {SUPPLIERS_CONFIG_PATH}")
except Exception as e:
log.error(f"Failed to save supplier settings file {SUPPLIERS_CONFIG_PATH}: {e}")
raise ConfigurationError(f"Failed to save supplier settings: {e}")

BIN
context_portal/conport_vector_data/3712b223-f80b-4c07-b57a-5cf7c8175c86/length.bin
View File

Binary file not shown.

BIN
context_portal/conport_vector_data/chroma.sqlite3
View File

Binary file not shown.

BIN
context_portal/context.db
View File

Binary file not shown.

388
gui/first_time_setup_dialog.py
View File

@ -1,388 +0,0 @@
import sys
import os
import shutil
import json
from pathlib import Path
from typing import Optional, Tuple
from PySide6.QtWidgets import (
QDialog, QVBoxLayout, QHBoxLayout, QLabel, QLineEdit, QPushButton,
QFileDialog, QMessageBox, QGroupBox, QFormLayout, QSpinBox, QDialogButtonBox
)
from PySide6.QtCore import Qt, Slot
# Constants for bundled resource locations relative to app base
BUNDLED_CONFIG_SUBDIR_NAME = "config"
BUNDLED_PRESETS_SUBDIR_NAME = "Presets"
DEFAULT_USER_DATA_SUBDIR_NAME = "user_data" # For portable path attempt
# Files to copy from bundled config to user config
DEFAULT_CONFIG_FILES = [
"asset_type_definitions.json",
"file_type_definitions.json",
"llm_settings.json",
"suppliers.json"
]
# app_settings.json is NOT copied. user_settings.json is handled separately.
USER_SETTINGS_FILENAME = "user_settings.json"
PERSISTENT_PATH_MARKER_FILENAME = ".first_run_complete"
PERSISTENT_CONFIG_ROOT_STORAGE_FILENAME = "asset_processor_user_root.txt" # Stores USER_CHOSEN_PATH
APP_NAME = "AssetProcessor" # Used for AppData paths
def get_app_base_dir() -> Path:
"""Determines the base directory for the application (executable or script)."""
if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
# Running in a PyInstaller bundle
return Path(sys._MEIPASS)
else:
# Running as a script
return Path(__file__).resolve().parent.parent # Assuming this file is in gui/ subdir
def get_os_specific_app_data_dir() -> Path:
"""Gets the OS-specific application data directory."""
if sys.platform == "win32":
path_str = os.getenv('APPDATA')
if path_str:
return Path(path_str) / APP_NAME
# Fallback if APPDATA is not set, though unlikely
return Path.home() / "AppData" / "Roaming" / APP_NAME
elif sys.platform == "darwin": # macOS
return Path.home() / "Library" / "Application Support" / APP_NAME
else: # Linux and other Unix-like
return Path.home() / ".config" / APP_NAME
class FirstTimeSetupDialog(QDialog):
def __init__(self, parent=None):
super().__init__(parent)
self.setWindowTitle("Asset Processor - First-Time Setup")
self.setModal(True)
self.setMinimumWidth(600)
self.app_base_dir = get_app_base_dir()
self.user_chosen_path: Optional[Path] = None
self._init_ui()
self._propose_default_config_path()
def _init_ui(self):
main_layout = QVBoxLayout(self)
# Configuration Path Group
config_path_group = QGroupBox("Configuration Location")
config_path_layout = QVBoxLayout()
self.proposed_path_label = QLabel("Proposed default configuration path:")
config_path_layout.addWidget(self.proposed_path_label)
path_selection_layout = QHBoxLayout()
self.config_path_edit = QLineEdit()
self.config_path_edit.setReadOnly(False) # Allow editing, then validate
path_selection_layout.addWidget(self.config_path_edit)
browse_button = QPushButton("Browse...")
browse_button.clicked.connect(self._browse_config_path)
path_selection_layout.addWidget(browse_button)
config_path_layout.addLayout(path_selection_layout)
config_path_group.setLayout(config_path_layout)
main_layout.addWidget(config_path_group)
# User Settings Group
user_settings_group = QGroupBox("Initial User Settings")
user_settings_form_layout = QFormLayout()
self.output_base_dir_edit = QLineEdit()
output_base_dir_browse_button = QPushButton("Browse...")
output_base_dir_browse_button.clicked.connect(self._browse_output_base_dir)
output_base_dir_layout = QHBoxLayout()
output_base_dir_layout.addWidget(self.output_base_dir_edit)
output_base_dir_layout.addWidget(output_base_dir_browse_button)
user_settings_form_layout.addRow("Default Library Output Path:", output_base_dir_layout)
self.output_dir_pattern_edit = QLineEdit("[supplier]/[asset_category]/[asset_name]")
user_settings_form_layout.addRow("Asset Structure Pattern:", self.output_dir_pattern_edit)
self.output_format_16bit_primary_edit = QLineEdit("png")
user_settings_form_layout.addRow("Default 16-bit Output Format (Primary):", self.output_format_16bit_primary_edit)
self.output_format_8bit_edit = QLineEdit("png")
user_settings_form_layout.addRow("Default 8-bit Output Format:", self.output_format_8bit_edit)
self.resolution_threshold_jpg_spinbox = QSpinBox()
self.resolution_threshold_jpg_spinbox.setRange(256, 16384)
self.resolution_threshold_jpg_spinbox.setValue(4096)
self.resolution_threshold_jpg_spinbox.setSuffix(" px")
user_settings_form_layout.addRow("JPG Resolution Threshold (for 8-bit):", self.resolution_threshold_jpg_spinbox)
user_settings_group.setLayout(user_settings_form_layout)
main_layout.addWidget(user_settings_group)
# Dialog Buttons
self.button_box = QDialogButtonBox(QDialogButtonBox.StandardButton.Ok | QDialogButtonBox.StandardButton.Cancel)
self.button_box.button(QDialogButtonBox.StandardButton.Ok).setText("Finish Setup")
self.button_box.accepted.connect(self._on_finish_setup)
self.button_box.rejected.connect(self.reject)
main_layout.addWidget(self.button_box)
def _propose_default_config_path(self):
proposed_path = None
# 1. Try portable path: user_data/ next to the application base dir
# If running from script, app_base_dir is .../Asset_processor_tool/gui, so parent is .../Asset_processor_tool
# If bundled, app_base_dir is the directory of the executable.
# Let's refine app_base_dir for portable path logic
# If script: Path(__file__).parent.parent = Asset_processor_tool
# If frozen: sys._MEIPASS (which is the temp extraction dir, not ideal for persistent user_data)
# A better approach for portable if frozen: Path(sys.executable).parent
current_app_dir = Path(sys.executable).parent if getattr(sys, 'frozen', False) else self.app_base_dir
portable_path_candidate = current_app_dir / DEFAULT_USER_DATA_SUBDIR_NAME
try:
portable_path_candidate.mkdir(parents=True, exist_ok=True)
if os.access(str(portable_path_candidate), os.W_OK):
proposed_path = portable_path_candidate
self.proposed_path_label.setText(f"Proposed portable path (writable):")
else:
self.proposed_path_label.setText(f"Portable path '{portable_path_candidate}' not writable.")
except Exception as e:
self.proposed_path_label.setText(f"Could not use portable path '{portable_path_candidate}': {e}")
print(f"Error checking/creating portable path: {e}") # For debugging
# 2. Fallback to OS-specific app data directory
if not proposed_path:
os_specific_path = get_os_specific_app_data_dir()
try:
os_specific_path.mkdir(parents=True, exist_ok=True)
if os.access(str(os_specific_path), os.W_OK):
proposed_path = os_specific_path
self.proposed_path_label.setText(f"Proposed standard path (writable):")
else:
self.proposed_path_label.setText(f"Standard path '{os_specific_path}' not writable. Please choose a location.")
except Exception as e:
self.proposed_path_label.setText(f"Could not use standard path '{os_specific_path}': {e}. Please choose a location.")
print(f"Error checking/creating standard path: {e}") # For debugging
if proposed_path:
self.config_path_edit.setText(str(proposed_path.resolve()))
else:
# Should not happen if OS specific path creation works, but as a last resort:
self.config_path_edit.setText(str(Path.home())) # Default to home if all else fails
QMessageBox.warning(self, "Path Issue", "Could not determine a default writable configuration path. Please select one manually.")
@Slot()
def _browse_config_path(self):
directory = QFileDialog.getExistingDirectory(
self,
"Select Configuration Directory",
self.config_path_edit.text() or str(Path.home())
)
if directory:
self.config_path_edit.setText(directory)
@Slot()
def _browse_output_base_dir(self):
directory = QFileDialog.getExistingDirectory(
self,
"Select Default Library Output Directory",
self.output_base_dir_edit.text() or str(Path.home())
)
if directory:
self.output_base_dir_edit.setText(directory)
def _validate_inputs(self) -> bool:
# Validate chosen config path
path_str = self.config_path_edit.text().strip()
if not path_str:
QMessageBox.warning(self, "Input Error", "Configuration path cannot be empty.")
return False
self.user_chosen_path = Path(path_str)
try:
self.user_chosen_path.mkdir(parents=True, exist_ok=True)
if not os.access(str(self.user_chosen_path), os.W_OK):
QMessageBox.warning(self, "Path Error", f"The chosen configuration path '{self.user_chosen_path}' is not writable.")
return False
except Exception as e:
QMessageBox.warning(self, "Path Error", f"Error with chosen configuration path '{self.user_chosen_path}': {e}")
return False
# Validate output base dir
output_base_dir_str = self.output_base_dir_edit.text().strip()
if not output_base_dir_str:
QMessageBox.warning(self, "Input Error", "Default Library Output Path cannot be empty.")
return False
try:
Path(output_base_dir_str).mkdir(parents=True, exist_ok=True) # Check if creatable
if not os.access(output_base_dir_str, os.W_OK):
QMessageBox.warning(self, "Path Error", f"The chosen output base path '{output_base_dir_str}' is not writable.")
return False
except Exception as e:
QMessageBox.warning(self, "Path Error", f"Error with output base path '{output_base_dir_str}': {e}")
return False
if not self.output_dir_pattern_edit.text().strip():
QMessageBox.warning(self, "Input Error", "Asset Structure Pattern cannot be empty.")
return False
if not self.output_format_16bit_primary_edit.text().strip():
QMessageBox.warning(self, "Input Error", "Default 16-bit Output Format cannot be empty.")
return False
if not self.output_format_8bit_edit.text().strip():
QMessageBox.warning(self, "Input Error", "Default 8-bit Output Format cannot be empty.")
return False
return True
def _copy_default_files(self):
if not self.user_chosen_path:
return
bundled_config_dir = self.app_base_dir / BUNDLED_CONFIG_SUBDIR_NAME
user_target_config_dir = self.user_chosen_path / BUNDLED_CONFIG_SUBDIR_NAME # User files also go into a 'config' subdir
try:
user_target_config_dir.mkdir(parents=True, exist_ok=True)
except Exception as e:
QMessageBox.critical(self, "Error", f"Could not create user config subdirectory '{user_target_config_dir}': {e}")
return
for filename in DEFAULT_CONFIG_FILES:
source_file = bundled_config_dir / filename
target_file = user_target_config_dir / filename
if not target_file.exists():
if source_file.is_file():
try:
shutil.copy2(str(source_file), str(target_file))
print(f"Copied '{source_file}' to '{target_file}'")
except Exception as e:
QMessageBox.warning(self, "File Copy Error", f"Could not copy '{filename}' to '{target_file}': {e}")
else:
print(f"Default config file '{source_file}' not found in bundle.")
else:
print(f"User config file '{target_file}' already exists. Skipping copy.")
# Copy Presets
bundled_presets_dir = self.app_base_dir / BUNDLED_PRESETS_SUBDIR_NAME
user_target_presets_dir = self.user_chosen_path / BUNDLED_PRESETS_SUBDIR_NAME
if bundled_presets_dir.is_dir():
try:
user_target_presets_dir.mkdir(parents=True, exist_ok=True)
for item in bundled_presets_dir.iterdir():
target_item = user_target_presets_dir / item.name
if not target_item.exists():
if item.is_file():
shutil.copy2(str(item), str(target_item))
print(f"Copied preset '{item.name}' to '{target_item}'")
# Add elif item.is_dir() for recursive copy if presets can have subdirs
except Exception as e:
QMessageBox.warning(self, "Preset Copy Error", f"Could not copy presets to '{user_target_presets_dir}': {e}")
else:
print(f"Bundled presets directory '{bundled_presets_dir}' not found.")
def _save_initial_user_settings(self):
if not self.user_chosen_path:
return
user_settings_path = self.user_chosen_path / USER_SETTINGS_FILENAME
settings_data = {}
# Load existing if it exists (though unlikely for first-time setup, but good practice)
if user_settings_path.exists():
try:
with open(user_settings_path, 'r', encoding='utf-8') as f:
settings_data = json.load(f)
except Exception as e:
QMessageBox.warning(self, "Error Loading Settings", f"Could not load existing user settings from '{user_settings_path}': {e}. Will create a new one.")
settings_data = {}
# Update with new values from dialog
settings_data['OUTPUT_BASE_DIR'] = self.output_base_dir_edit.text().strip()
settings_data['OUTPUT_DIRECTORY_PATTERN'] = self.output_dir_pattern_edit.text().strip()
settings_data['OUTPUT_FORMAT_16BIT_PRIMARY'] = self.output_format_16bit_primary_edit.text().strip().lower()
settings_data['OUTPUT_FORMAT_8BIT'] = self.output_format_8bit_edit.text().strip().lower()
settings_data['RESOLUTION_THRESHOLD_FOR_JPG'] = self.resolution_threshold_jpg_spinbox.value()
# Ensure general_settings exists for app_version if needed, or other core settings
if 'general_settings' not in settings_data:
settings_data['general_settings'] = {}
# Example: settings_data['general_settings']['some_new_user_setting'] = True
try:
with open(user_settings_path, 'w', encoding='utf-8') as f:
json.dump(settings_data, f, indent=4)
print(f"Saved user settings to '{user_settings_path}'")
except Exception as e:
QMessageBox.critical(self, "Error Saving Settings", f"Could not save user settings to '{user_settings_path}': {e}")
def _save_persistent_info(self):
if not self.user_chosen_path:
return
# 1. Save USER_CHOSEN_PATH to a persistent location (e.g., AppData)
persistent_storage_dir = get_os_specific_app_data_dir()
try:
persistent_storage_dir.mkdir(parents=True, exist_ok=True)
persistent_path_file = persistent_storage_dir / PERSISTENT_CONFIG_ROOT_STORAGE_FILENAME
with open(persistent_path_file, 'w', encoding='utf-8') as f:
f.write(str(self.user_chosen_path.resolve()))
print(f"Saved chosen config path to '{persistent_path_file}'")
except Exception as e:
QMessageBox.warning(self, "Error Saving Path", f"Could not persistently save the chosen configuration path: {e}")
# This is not critical enough to stop the setup, but user might need to re-select on next launch.
# 2. Create marker file in USER_CHOSEN_PATH
marker_file = self.user_chosen_path / PERSISTENT_PATH_MARKER_FILENAME
try:
with open(marker_file, 'w', encoding='utf-8') as f:
f.write("Asset Processor first-time setup complete.")
print(f"Created marker file at '{marker_file}'")
except Exception as e:
QMessageBox.warning(self, "Error Creating Marker", f"Could not create first-run marker file at '{marker_file}': {e}")
@Slot()
def _on_finish_setup(self):
if not self._validate_inputs():
return
# Confirmation before proceeding
reply = QMessageBox.question(self, "Confirm Setup",
f"The following path will be used for configuration and user data:\n"
f"{self.user_chosen_path}\n\n"
f"Default configuration files and presets will be copied if they don't exist.\n"
f"Initial user settings will be saved.\n\nProceed with setup?",
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No)
if reply == QMessageBox.StandardButton.No:
return
try:
self._copy_default_files()
self._save_initial_user_settings()
self._save_persistent_info()
QMessageBox.information(self, "Setup Complete", "First-time setup completed successfully!")
self.accept()
except Exception as e:
QMessageBox.critical(self, "Setup Error", f"An unexpected error occurred during setup: {e}")
# Optionally, attempt cleanup or guide user
def get_chosen_config_path(self) -> Optional[Path]:
"""Returns the path chosen by the user after successful completion."""
if self.result() == QDialog.DialogCode.Accepted:
return self.user_chosen_path
return None
if __name__ == '__main__':
from PySide6.QtWidgets import QApplication
app = QApplication(sys.argv)
dialog = FirstTimeSetupDialog()
if dialog.exec():
chosen_path = dialog.get_chosen_config_path()
print(f"Dialog accepted. Chosen config path: {chosen_path}")
else:
print("Dialog cancelled.")
sys.exit()

206
main.py
View File

@ -15,12 +15,11 @@ from typing import List, Dict, Tuple, Optional
# --- Utility Imports ---
from utils.hash_utils import calculate_sha256
from utils.path_utils import get_next_incrementing_value
from utils import app_setup_utils # Import the new utility module
# --- Qt Imports for Application Structure ---
from PySide6.QtCore import QObject, Slot, QThreadPool, QRunnable, Signal
from PySide6.QtCore import Qt
from PySide6.QtWidgets import QApplication, QDialog # Import QDialog for the setup dialog
from PySide6.QtWidgets import QApplication
# --- Backend Imports ---
# Add current directory to sys.path for direct execution
@ -46,10 +45,6 @@ try:
from gui.main_window import MainWindow
print("DEBUG: Successfully imported MainWindow.")
print("DEBUG: Attempting to import FirstTimeSetupDialog...")
from gui.first_time_setup_dialog import FirstTimeSetupDialog # Import the setup dialog
print("DEBUG: Successfully imported FirstTimeSetupDialog.")
print("DEBUG: Attempting to import prepare_processing_workspace...")
from utils.workspace_utils import prepare_processing_workspace
print("DEBUG: Successfully imported prepare_processing_workspace.")
@ -305,9 +300,8 @@ class App(QObject):
# Signal emitted when all queued processing tasks are complete
all_tasks_finished = Signal(int, int, int) # processed_count, skipped_count, failed_count (Placeholder counts for now)
def __init__(self, user_config_path: str):
def __init__(self):
super().__init__()
self.user_config_path = user_config_path # Store the determined user config path
self.config_obj = None
self.processing_engine = None
self.main_window = None
@ -316,19 +310,34 @@ class App(QObject):
self._task_results = {"processed": 0, "skipped": 0, "failed": 0}
log.info(f"Maximum threads for pool: {self.thread_pool.maxThreadCount()}")
self._load_config(self.user_config_path) # Pass the user config path
self._load_config()
self._init_engine()
self._init_gui()
def _load_config(self, user_config_path: str):
"""Loads the base configuration using the determined user config path."""
def _load_config(self):
"""Loads the base configuration using a default preset."""
# The actual preset name comes from the GUI request later, but the engine
# needs an initial valid configuration object.
try:
# Initialize Configuration with the determined user config path
# The Configuration class is responsible for finding presets and other configs
self.config_obj = Configuration(base_dir_user_config=user_config_path)
log.info(f"Base configuration loaded using user config path: '{user_config_path}'.")
# Find the first available preset to use as a default
preset_dir = Path(__file__).parent / "Presets"
default_preset_name = None
if preset_dir.is_dir():
presets = sorted([f.stem for f in preset_dir.glob("*.json") if f.is_file() and not f.name.startswith('_')])
if presets:
default_preset_name = presets[0]
log.info(f"Using first available preset as default for initial config: '{default_preset_name}'")
if not default_preset_name:
# Fallback or raise error if no presets found
log.error("No presets found in the 'Presets' directory. Cannot initialize default configuration.")
# Option 1: Raise an error
raise ConfigurationError("No presets found to load default configuration.")
self.config_obj = Configuration(preset_name=default_preset_name)
log.info(f"Base configuration loaded using default preset '{default_preset_name}'.")
except ConfigurationError as e:
log.error(f"Fatal: Failed to load base configuration using user config path '{user_config_path}': {e}")
log.error(f"Fatal: Failed to load base configuration using default preset: {e}")
# In a real app, show this error to the user before exiting
sys.exit(1)
except Exception as e:
@ -392,6 +401,120 @@ class App(QObject):
log.debug(f"Initialized active task count to: {self._active_tasks_count}")
# Update GUI progress bar/status via MainPanelWidget
self.main_window.main_panel_widget.progress_bar.setMaximum(len(source_rules))
self.main_window.main_panel_widget.progress_bar.setValue(0)
self.main_window.main_panel_widget.progress_bar.setFormat(f"0/{len(source_rules)} tasks")
# --- Get paths needed for ProcessingTask ---
try:
# Get output_dir from processing_settings passed from autotest.py
output_base_path_str = processing_settings.get("output_dir")
log.info(f"APP_DEBUG: Received output_dir in processing_settings: {output_base_path_str}")
if not output_base_path_str:
log.error("Cannot queue tasks: Output directory path is empty in processing_settings.")
# self.main_window.statusBar().showMessage("Error: Output directory cannot be empty.", 5000) # GUI specific
return
output_base_path = Path(output_base_path_str)
# Basic validation - check if it's likely a valid path structure (doesn't guarantee existence/writability here)
if not output_base_path.is_absolute():
# Or attempt to resolve relative to workspace? For now, require absolute from GUI.
log.warning(f"Output path '{output_base_path}' is not absolute. Processing might fail if relative path is not handled correctly by engine.")
# Consider resolving: output_base_path = Path.cwd() / output_base_path # If relative paths are allowed
# Define workspace path (assuming main.py is in the project root)
workspace_path = Path(__file__).parent.resolve()
log.debug(f"Using Workspace Path: {workspace_path}")
log.debug(f"Using Output Base Path: {output_base_path}")
except Exception as e:
log.exception(f"Error getting/validating paths for processing task: {e}")
self.main_window.statusBar().showMessage(f"Error preparing paths: {e}", 5000)
return
# --- End Get paths ---
# Set max threads based on GUI setting
worker_count = processing_settings.get('workers', 1)
self.thread_pool.setMaxThreadCount(worker_count)
log.info(f"Set thread pool max workers to: {worker_count}")
# Queue tasks in the thread pool
log.debug("DEBUG: Entering task queuing loop.")
for i, rule in enumerate(source_rules):
if isinstance(rule, SourceRule):
log.info(f"DEBUG Task {i+1}: Rule Input='{rule.input_path}', Supplier ID='{getattr(rule, 'supplier_identifier', 'Not Set')}', Preset='{getattr(rule, 'preset_name', 'Not Set')}'")
log.debug(f"DEBUG: Preparing to queue task {i+1}/{len(source_rules)} for rule: {rule.input_path}")
# --- Create a new Configuration and Engine instance for this specific task ---
task_engine = None
try:
# Get preset name from the rule, fallback to app's default if missing
preset_name_for_task = getattr(rule, 'preset_name', None)
if not preset_name_for_task:
log.warning(f"Task {i+1} (Rule: {rule.input_path}): SourceRule missing preset_name. Falling back to default preset '{self.config_obj.preset_name}'.")
preset_name_for_task = self.config_obj.preset_name
task_config = Configuration(preset_name=preset_name_for_task)
task_engine = ProcessingEngine(task_config)
log.debug(f"Task {i+1}: Created new ProcessingEngine instance with preset '{preset_name_for_task}'.")
except ConfigurationError as config_err:
log.error(f"Task {i+1} (Rule: {rule.input_path}): Failed to load configuration for preset '{preset_name_for_task}': {config_err}. Skipping task.")
self._active_tasks_count -= 1 # Decrement count as this task won't run
self._task_results["failed"] += 1
# Optionally update GUI status for this specific rule
self.main_window.update_file_status(str(rule.input_path), "failed", f"Config Error: {config_err}")
continue # Skip to the next rule
except Exception as engine_err:
log.exception(f"Task {i+1} (Rule: {rule.input_path}): Failed to initialize ProcessingEngine for preset '{preset_name_for_task}': {engine_err}. Skipping task.")
self._active_tasks_count -= 1 # Decrement count
self._task_results["failed"] += 1
self.main_window.update_file_status(str(rule.input_path), "failed", f"Engine Init Error: {engine_err}")
continue # Skip to the next rule
if task_engine is None: # Should not happen if exceptions are caught, but safety check
log.error(f"Task {i+1} (Rule: {rule.input_path}): Engine is None after initialization attempt. Skipping task.")
self._active_tasks_count -= 1 # Decrement count
self._task_results["failed"] += 1
self.main_window.update_file_status(str(rule.input_path), "failed", "Engine initialization failed (unknown reason).")
continue # Skip to the next rule
# --- End Engine Instantiation ---
task = ProcessingTask(
engine=task_engine,
rule=rule,
workspace_path=workspace_path,
output_base_path=output_base_path # This is Path(output_base_path_str)
)
log.info(f"APP_DEBUG: Passing to ProcessingTask: output_base_path = {output_base_path}")
task.signals.finished.connect(self._on_task_finished)
log.debug(f"DEBUG: Calling thread_pool.start() for task {i+1}")
self.thread_pool.start(task)
log.debug(f"DEBUG: Returned from thread_pool.start() for task {i+1}")
else:
log.warning(f"Skipping invalid item (index {i}) in rule list: {type(rule)}")
log.info(f"Queued {len(source_rules)} processing tasks (finished loop).")
# GUI status already updated in MainWindow when signal was emitted
# --- Slot to handle completion of individual tasks ---
@Slot(str, str, object)
def _on_task_finished(self, rule_input_path, status, result_or_error):
"""Handles the 'finished' signal from a ProcessingTask."""
log.info(f"Task finished signal received for {rule_input_path}. Status: {status}")
self._active_tasks_count -= 1
log.debug(f"Active tasks remaining: {self._active_tasks_count}")
# Update overall results (basic counts for now)
if status == "processed":
self._task_results["processed"] += 1
elif status == "skipped": # Assuming engine might return 'skipped' status eventually
self._task_results["skipped"] += 1
else: # Count all other statuses (failed_preparation, failed_processing) as failed
self._task_results["failed"] += 1
# Update progress bar via MainPanelWidget
total_tasks = self.main_window.main_panel_widget.progress_bar.maximum()
completed_tasks = total_tasks - self._active_tasks_count
self.main_window.main_panel_widget.update_progress_bar(completed_tasks, total_tasks) # Use MainPanelWidget's method
@ -435,56 +558,9 @@ if __name__ == "__main__":
log.info("No required CLI arguments detected, starting GUI mode.")
# --- Run the GUI Application ---
try:
user_config_path = app_setup_utils.read_saved_user_config_path()
log.debug(f"Read saved user config path: {user_config_path}")
qt_app = QApplication(sys.argv)
first_run_needed = False
if user_config_path is None or not user_config_path.strip():
log.info("No saved user config path found. First run setup needed.")
first_run_needed = True
else:
user_config_dir = Path(user_config_path)
marker_file = app_setup_utils.get_first_run_marker_file(user_config_path)
if not user_config_dir.is_dir():
log.warning(f"Saved user config directory does not exist: {user_config_path}. First run setup needed.")
first_run_needed = True
elif not Path(marker_file).is_file():
log.warning(f"First run marker file not found in {user_config_path}. First run setup needed.")
first_run_needed = True
else:
log.info(f"Saved user config path found and valid: {user_config_path}. Marker file exists.")
qt_app = None
if first_run_needed:
log.info("Initiating first-time setup dialog.")
# Need a QApplication instance to show the dialog
qt_app = QApplication.instance()
if qt_app is None:
qt_app = QApplication(sys.argv)
dialog = FirstTimeSetupDialog()
if dialog.exec() == QDialog.Accepted:
user_config_path = dialog.get_chosen_path()
log.info(f"First-time setup completed. Chosen path: {user_config_path}")
# The dialog should have already saved the path and created the marker file
else:
log.info("First-time setup cancelled by user. Exiting application.")
sys.exit(0) # Exit gracefully
# If qt_app was created for the dialog, reuse it. Otherwise, create it now.
if qt_app is None:
qt_app = QApplication.instance()
if qt_app is None:
qt_app = QApplication(sys.argv)
# Ensure user_config_path is set before initializing App
if not user_config_path or not Path(user_config_path).is_dir():
log.error(f"Fatal: User config path is invalid or not set after setup: {user_config_path}. Cannot proceed.")
sys.exit(1)
app_instance = App(user_config_path) # Pass the determined path
app_instance = App()
app_instance.run()
sys.exit(qt_app.exec())

66
utils/app_setup_utils.py
View File

@ -1,66 +0,0 @@
import os
import sys
import platform
def get_app_data_dir():
"""
Gets the OS-specific application data directory for Asset Processor.
Uses standard library methods as appdirs is not available.
"""
app_name = "AssetProcessor"
if platform.system() == "Windows":
# On Windows, use APPDATA environment variable
app_data_dir = os.path.join(os.environ.get("APPDATA", "~"), app_name)
elif platform.system() == "Darwin":
# On macOS, use ~/Library/Application Support
app_data_dir = os.path.join("~", "Library", "Application Support", app_name)
else:
# On Linux and other Unix-like systems, use ~/.config
app_data_dir = os.path.join("~", ".config", app_name)
# Expand the user home directory symbol if present
return os.path.expanduser(app_data_dir)
def get_persistent_config_path_file():
"""
Gets the full path to the file storing the user's chosen config directory.
"""
app_data_dir = get_app_data_dir()
# Ensure the app data directory exists
os.makedirs(app_data_dir, exist_ok=True)
return os.path.join(app_data_dir, "asset_processor_user_root.txt")
def read_saved_user_config_path():
"""
Reads the saved user config path from the persistent file.
Returns the path string or None if the file doesn't exist or is empty.
"""
path_file = get_persistent_config_path_file()
if os.path.exists(path_file):
try:
with open(path_file, "r", encoding="utf-8") as f:
saved_path = f.read().strip()
if saved_path:
return saved_path
except IOError:
# Handle potential file reading errors
pass
return None
def save_user_config_path(user_config_path):
"""
Saves the user's chosen config path to the persistent file.
"""
path_file = get_persistent_config_path_file()
try:
with open(path_file, "w", encoding="utf-8") as f:
f.write(user_config_path)
except IOError:
# Handle potential file writing errors
print(f"Error saving user config path to {path_file}", file=sys.stderr)
def get_first_run_marker_file(user_config_path):
"""
Gets the full path to the first-run marker file within the user config directory.
"""
return os.path.join(user_config_path, ".first_run_complete")