Asset-Frameworker/Documentation/02_Developer_Guide/01_Architecture.md

Developer Guide: Architecture

This document provides a high-level overview of the Asset Processor Tool's architecture and core components for developers.

High-Level Architecture

The Asset Processor Tool is designed to process 3D asset source files into a standardized library format. Its high-level architecture consists of:

1. Core Processing Engine (processing_engine.py): The primary component responsible for executing the asset processing pipeline for a single input asset based on a provided SourceRule object and static configuration. The previous asset_processor.py has been removed.
2. Prediction System: Responsible for analyzing input files and generating the initial SourceRule hierarchy with predicted values. This system utilizes a base handler (gui/base_prediction_handler.py::BasePredictionHandler) with specific implementations:
   • Rule-Based Predictor (gui/prediction_handler.py::RuleBasedPredictionHandler): Uses predefined rules from presets to classify files and determine initial processing parameters.
   • LLM Predictor (gui/llm_prediction_handler.py::LLMPredictionHandler): An experimental alternative that uses a Large Language Model (LLM) to interpret file contents and context to predict processing parameters.
3. Configuration System (Configuration): Handles loading core settings (including centralized type definitions and LLM-specific configuration) and merging them with supplier-specific rules defined in JSON presets and the persistent config/suppliers.json file.
4. Multiple Interfaces: Provides different ways to interact with the tool:
   • Graphical User Interface (GUI)
   • Command-Line Interface (CLI) - Note: The primary CLI execution logic (run_cli in main.py) is currently non-functional/commented out post-refactoring.
   • Directory Monitor for automated processing. The GUI acts as the primary source of truth for processing rules, coordinating the generation and management of the SourceRule hierarchy before sending it to the processing engine. It accumulates prediction results from multiple input sources before updating the view. The Monitor interface can also generate SourceRule objects (using utils/prediction_utils.py) to bypass the GUI for automated workflows.
5. Optional Integration: Includes scripts (blenderscripts/) for integrating with Blender. Logic for executing these scripts was intended to be centralized in utils/blender_utils.py, but this utility has not yet been implemented.

Hierarchical Rule System

A key addition to the architecture is the Hierarchical Rule System, which provides a dynamic layer of configuration that can override the static settings loaded from presets. This system is represented by a hierarchy of rule objects: SourceRule, AssetRule, and FileRule.

• SourceRule: Represents rules applied at the top level, typically corresponding to an entire input source (e.g., a ZIP file or folder).
• AssetRule: Represents rules applied to a specific asset within a source (a source can contain multiple assets).
• FileRule: Represents rules applied to individual files within an asset.

This hierarchy allows for fine-grained control over processing parameters. The GUI's prediction logic generates this hierarchy with initial predicted values for overridable fields based on presets and file analysis. The processing engine then operates solely on the explicit values provided in this SourceRule object and static configuration, without internal prediction or fallback logic.

Core Components

• config/app_settings.json: Defines core, global settings, constants, and centralized definitions for allowed asset and file types (ASSET_TYPE_DEFINITIONS, FILE_TYPE_DEFINITIONS), including metadata like colors and descriptions. This replaces the old config.py file.
• config/suppliers.json: A persistent JSON file storing known supplier names for GUI auto-completion.
• Presets/*.json: Supplier-specific JSON files defining rules for file interpretation and initial prediction.
• configuration.py (Configuration class): Loads config/app_settings.json settings and merges them with a selected preset, pre-compiling regex patterns for efficiency. This static configuration is used by the processing engine.
• rule_structure.py: Defines the SourceRule, AssetRule, and FileRule dataclasses used to represent the hierarchical processing rules.
• gui/: Directory containing modules for the Graphical User Interface (GUI), built with PySide6. The MainWindow (main_window.py) acts as a coordinator, orchestrating interactions between various components. Key GUI components include:
  • main_panel_widget.py::MainPanelWidget: Contains the primary controls for loading sources, selecting presets, viewing/editing rules, and initiating processing.
  • preset_editor_widget.py::PresetEditorWidget: Provides the interface for managing presets.
  • log_console_widget.py::LogConsoleWidget: Displays application logs.
  • unified_view_model.py::UnifiedViewModel: Implements the QAbstractItemModel for the hierarchical rule view, holding SourceRule data and managing display logic (coloring, etc.). Caches configuration data for performance.
  • rule_hierarchy_model.py::RuleHierarchyModel: A simpler model used internally by the UnifiedViewModel to manage the SourceRule data structure.
  • delegates.py: Contains custom QStyledItemDelegate implementations for inline editing in the rule view.
  • asset_restructure_handler.py::AssetRestructureHandler: Handles complex model updates when a file's target asset is changed via the GUI, ensuring the SourceRule hierarchy is correctly modified.
  • base_prediction_handler.py::BasePredictionHandler: Abstract base class for prediction logic.
  • prediction_handler.py::RuleBasedPredictionHandler: Generates the initial SourceRule hierarchy based on presets and file analysis. Inherits from BasePredictionHandler.
  • llm_prediction_handler.py::LLMPredictionHandler: Experimental predictor using an LLM. Inherits from BasePredictionHandler.
  • llm_interaction_handler.py::LLMInteractionHandler: Manages communication with the LLM service for the LLM predictor.
• processing_engine.py (ProcessingEngine class): The core component that executes the processing pipeline for a single SourceRule object using the static Configuration. A new instance is created per task for state isolation.
• main.py: The main entry point for the application. Primarily launches the GUI. Contains commented-out/non-functional CLI logic (run_cli).
• monitor.py: Implements the directory monitoring feature using watchdog. It now processes archives asynchronously using a ThreadPoolExecutor, leveraging utils.prediction_utils.py for rule generation and utils.workspace_utils.py for workspace management before invoking the ProcessingEngine.
• blenderscripts/: Contains Python scripts designed to be executed within Blender for post-processing tasks.
• utils/: Directory containing utility modules:
  • workspace_utils.py: Contains functions like prepare_processing_workspace for handling temporary directories and archive extraction.
  • prediction_utils.py: Contains functions like generate_source_rule_from_archive used by the monitor for rule-based prediction.
  • blender_utils.py: (Intended location for Blender script execution logic, currently not implemented).

Processing Pipeline (Simplified)

The primary processing engine (processing_engine.py) executes a series of steps for each asset based on the provided SourceRule object and static configuration:

1. Extraction of input to a temporary workspace (using utils.workspace_utils.py).
2. Classification of files (map, model, extra, ignored, unrecognised) based only on the provided SourceRule object (classification/prediction happens before the engine is called).
3. Determination of base metadata (asset name, category, archetype).
4. Skip check if output exists and overwrite is not forced.
5. Processing of maps (resize, format/bit depth conversion, inversion, stats calculation).
6. Merging of channels based on rules.
7. Generation of metadata.json file.
8. Organization of processed files into the final output structure.
9. Cleanup of the temporary workspace.
10. (Optional) Execution of Blender scripts (currently triggered directly, intended to use utils.blender_utils.py).

This architecture allows for a modular design, separating configuration, rule generation/management (GUI, Monitor utilities), and core processing execution. The SourceRule object serves as a clear data contract between the rule generation layer and the processing engine. Parallel processing (in Monitor) and background threads (in GUI) are utilized for efficiency and responsiveness.