Asset-Frameworker/Deprecated/detailed_documentation_plan.md

Final Plan for Updating documentation.txt with Debugging Details

This document outlines the final plan for enhancing documentation.txt with detailed internal information crucial for debugging the Asset Processor Tool. This plan incorporates analysis of asset_processor.py, configuration.py, main.py, and gui/processing_handler.py.

Task Objective

Analyze relevant source code (asset_processor.py, configuration.py, main.py, gui/processing_handler.py) to gather specific details about internal logic, state management, error handling, data structures, concurrency, resource management, and limitations. Integrate this information into the existing documentation.txt to aid developers in debugging.

Analysis Steps Completed

1. Read and analyzed readme.md.
2. Listed code definitions in the root directory (.).
3. Listed code definitions in the gui/ directory.
4. Read and analyzed asset_processor.py.
5. Read and analyzed configuration.py.
6. Read and analyzed main.py.
7. Read and analyzed gui/processing_handler.py.

Final Integration Plan (Merged Structure)

1. Add New Top-Level Section:

   • Append the following section header to the end of documentation.txt:

     ================================
     Internal Details for Debugging
     ================================
     

2. Create Subsections:

   • Under the new "Internal Details" section, create the following subsections (renumbering from 7 onwards):
     • 7. Internal Logic & Algorithms
     • 8. State Management
     • 9. Error Handling & Propagation
     • 10. Key Data Structures
     • 11. Concurrency Models (CLI & GUI)
     • 12. Resource Management
     • 13. Known Limitations & Edge Cases

3. Populate Subsections with Specific Details:

   • 7. Internal Logic & Algorithms:

     • Configuration Preparation (Configuration class): Detail the __init__ process: loading config.py and preset JSON, validating structure (_validate_configs), compiling regex (_compile_regex_patterns using _fnmatch_to_regex). Mention compiled patterns storage.
     • CLI Argument Parsing (main.py:setup_arg_parser): Briefly describe argparse usage and key flags influencing execution.
     • Output Directory Resolution (main.py:main): Explain how the final output path is determined and resolved.
     • Asset Processing (AssetProcessor class):
       • Classification (_inventory_and_classify_files): Describe multi-pass approach using compiled regex from Configuration. Detail variant sorting criteria.
       • Map Processing (_process_maps): Detail image loading, Gloss->Rough inversion, resizing, format determination (using Configuration rules), bit depth conversion, stats calculation, aspect ratio logic, save fallback.
       • Merging (_merge_maps): Detail common resolution finding, input loading, channel merging (using Configuration rules), output determination, save fallback.
       • Metadata (_determine_base_metadata, etc.): Summarize base name extraction, category/archetype determination (using Configuration rules), metadata.json population.
     • Blender Integration (main.py:run_blender_script, gui/processing_handler.py:_run_blender_script_subprocess): Explain subprocess execution, command construction (-b, --python, --), argument passing (asset_root_dir).

   • 8. State Management:

     • Configuration object: Holds loaded config state and compiled regex. Instantiated per worker.
     • AssetProcessor: Primarily stateless between process() calls. Internal state within process() managed by local variables (e.g., current_asset_metadata). self.classified_files populated once and filtered per asset.
     • main.py: Tracks overall CLI run counts (processed, skipped, failed).
     • gui/processing_handler.py: Manages GUI processing run state via flags (_is_running, _cancel_requested) and stores Future objects (_futures).

   • 9. Error Handling & Propagation:

     • Custom Exceptions: AssetProcessingError, ConfigurationError.
     • Configuration: Raises ConfigurationError on load/validation failure. Logs regex compilation warnings.
     • AssetProcessor: Catches exceptions within per-asset loop, logs error, marks asset "failed", continues loop. Handles specific save fallbacks (EXR->PNG). Raises AssetProcessingError for critical setup failures.
     • Worker Wrapper (main.py:process_single_asset_wrapper): Catches exceptions from Configuration/AssetProcessor, logs, returns "failed" status tuple.
     • Process Pool (main.py, gui/processing_handler.py): try...except around executor block catches pool-level errors.
     • GUI Communication (ProcessingHandler): Catches errors during future result retrieval, emits failure status via signals.
     • Blender Scripts: Checks subprocess return code, logs stderr. Catches FileNotFoundError.

   • 10. Key Data Structures:

     • Configuration attributes: compiled_map_keyword_regex, compiled_extra_regex, etc. (compiled re.Pattern objects).
     • AssetProcessor structures: self.classified_files (dict[str, list[dict]]), processed_maps_details_asset (dict[str, dict[str, dict]]), file_to_base_name_map (dict[Path, Optional[str]]).
     • Return values: Status dictionary from AssetProcessor.process(), status tuple from process_single_asset_wrapper.
     • ProcessingHandler._futures: dict[Future, str].

   • 11. Concurrency Models (CLI & GUI):

     • Common Core: Both use concurrent.futures.ProcessPoolExecutor running main.process_single_asset_wrapper. Configuration and AssetProcessor instantiated within each worker process for isolation.
     • CLI Orchestration (main.py:run_processing): Direct executor usage, as_completed gathers results synchronously.
     • GUI Orchestration (gui/processing_handler.py): ProcessingHandler (QObject) runs executor logic in a QThread. Results processed in handler thread, communicated asynchronously to UI thread via Qt signals (progress_updated, file_status_updated, processing_finished).
     • Cancellation (gui/processing_handler.py:request_cancel): Sets flag, attempts executor.shutdown(wait=False), tries cancelling pending futures. Limitation: Does not stop already running workers.

   • 12. Resource Management:

     • Configuration: Uses with open for preset files.
     • AssetProcessor: Manages temporary workspace (tempfile.mkdtemp, shutil.rmtree in finally). Uses with open for metadata JSON.
     • ProcessPoolExecutor: Lifecycle managed via with statement in main.py and gui/processing_handler.py.

   • 13. Known Limitations & Edge Cases:

     • Configuration: Basic structural validation only; regex compilation errors are warnings; _fnmatch_to_regex helper is basic.
     • AssetProcessor: Relies heavily on filename patterns; high memory potential for large images; limited intra-asset error recovery; simplified prediction logic.
     • CLI: Basic preset file existence check only before starting workers; Blender executable finding logic order (config > PATH).
     • GUI Concurrency: Cancellation doesn't stop currently executing worker processes.

4. Switch Mode: Request switching to Code mode to apply these changes by appending to documentation.txt.