Asset-Frameworker/Documentation/02_Developer_Guide/05_Processing_Pipeline.md

Developer Guide: Processing Pipeline

This document details the step-by-step technical process executed by the asset processing pipeline, which is initiated by the ProcessingEngine class (processing_engine.py) and orchestrated by the PipelineOrchestrator (processing/pipeline/orchestrator.py).

The ProcessingEngine.process() method serves as the main entry point. It initializes a PipelineOrchestrator instance, providing it with the application's Configuration object and a predefined list of processing stages. The PipelineOrchestrator.process_source_rule() method then manages the execution of these stages for each asset defined in the input SourceRule.

A crucial component in this architecture is the AssetProcessingContext (processing/pipeline/asset_context.py). An instance of this dataclass is created for each AssetRule being processed. It acts as a stateful container, carrying all relevant data (source files, rules, configuration, intermediate results, metadata) and is passed sequentially through each stage. Each stage can read from and write to the context, allowing data to flow and be modified throughout the pipeline.

The pipeline stages are executed in the following order:

1. SupplierDeterminationStage (processing/pipeline/stages/supplier_determination.py):

   • Responsibility: Determines the effective supplier for the asset based on the SourceRule's supplier_identifier, supplier_override, and supplier definitions in the Configuration.
   • Context Interaction: Updates AssetProcessingContext.effective_supplier and potentially AssetProcessingContext.asset_metadata with supplier information.

2. AssetSkipLogicStage (processing/pipeline/stages/asset_skip_logic.py):

   • Responsibility: Checks if the asset should be skipped, typically if the output already exists and overwriting is not forced.
   • Context Interaction: Sets AssetProcessingContext.status_flags['skip_asset'] to True if the asset should be skipped, halting further processing for this asset by the orchestrator.

3. MetadataInitializationStage (processing/pipeline/stages/metadata_initialization.py):

   • Responsibility: Initializes the AssetProcessingContext.asset_metadata dictionary with base information derived from the AssetRule, SourceRule, and Configuration. This includes asset name, type, and any common metadata.
   • Context Interaction: Populates AssetProcessingContext.asset_metadata.

4. FileRuleFilterStage (processing/pipeline/stages/file_rule_filter.py):

   • Responsibility: Filters the FileRule objects from the AssetRule to determine which files should actually be processed. It respects FILE_IGNORE rules.
   • Context Interaction: Populates AssetProcessingContext.files_to_process with the list of FileRule objects that passed the filter.

5. GlossToRoughConversionStage (processing/pipeline/stages/gloss_to_rough_conversion.py):

   • Responsibility: Identifies gloss maps (based on FileRule properties and filename conventions) that are intended to be used as roughness maps. If found, it loads the image, inverts its colors, and saves a temporary inverted version.
   • Context Interaction: Modifies FileRule objects in AssetProcessingContext.files_to_process (e.g., updates file_path to point to the temporary inverted map, sets flags indicating inversion). Updates AssetProcessingContext.processed_maps_details with information about the conversion.

6. AlphaExtractionToMaskStage (processing/pipeline/stages/alpha_extraction_to_mask.py):

   • Responsibility: If a FileRule specifies alpha channel extraction (e.g., from a diffuse map to create an opacity mask), this stage loads the source image, extracts its alpha channel, and saves it as a new temporary grayscale map.
   • Context Interaction: May add new FileRule-like entries or details to AssetProcessingContext.processed_maps_details representing the extracted mask.

7. NormalMapGreenChannelStage (processing/pipeline/stages/normal_map_green_channel.py):

   • Responsibility: Checks FileRules for normal maps and, based on configuration (e.g., invert_normal_map_green_channel for a specific supplier), potentially inverts the green channel of the normal map image.
   • Context Interaction: Modifies the image data for normal maps if inversion is needed, saving a new temporary version. Updates AssetProcessingContext.processed_maps_details.

8. IndividualMapProcessingStage (processing/pipeline/stages/individual_map_processing.py):

   • Responsibility: Processes individual texture map files. This includes:
     • Loading the source image.
     • Applying Power-of-Two (POT) scaling.
     • Generating multiple resolution variants based on configuration.
     • Handling color space conversions (e.g., BGR to RGB).
     • Calculating image statistics (min, max, mean, median).
     • Determining and storing aspect ratio change information.
     • Saving processed temporary map files.
     • Applying name variant suffixing and using standard type aliases for filenames.
   • Context Interaction: Heavily populates AssetProcessingContext.processed_maps_details with paths to temporary processed files, dimensions, and other metadata for each map and its variants. Updates AssetProcessingContext.asset_metadata with image stats and aspect ratio info.

9. MapMergingStage (processing/pipeline/stages/map_merging.py):

   • Responsibility: Performs channel packing and other merge operations based on map_merge_rules defined in the Configuration.
   • Context Interaction: Reads source map details and temporary file paths from AssetProcessingContext.processed_maps_details. Saves new temporary merged maps and records their details in AssetProcessingContext.merged_maps_details.

10. MetadataFinalizationAndSaveStage (processing/pipeline/stages/metadata_finalization_save.py):

    • Responsibility: Collects all accumulated metadata from AssetProcessingContext.asset_metadata, AssetProcessingContext.processed_maps_details, and AssetProcessingContext.merged_maps_details. It structures this information and saves it as the metadata.json file in a temporary location within the engine's temporary directory.
    • Context Interaction: Reads from various context fields and writes the metadata.json file. Stores the path to this temporary metadata file in the context (e.g., AssetProcessingContext.asset_metadata['temp_metadata_path']).

11. OutputOrganizationStage (processing/pipeline/stages/output_organization.py):

    • Responsibility: Determines final output paths for all processed maps, merged maps, the metadata file, and any other asset files (like models). It then copies these files from their temporary locations to the final structured output directory.
    • Context Interaction: Reads temporary file paths from AssetProcessingContext.processed_maps_details, AssetProcessingContext.merged_maps_details, and the temporary metadata file path. Uses Configuration for output path patterns. Updates AssetProcessingContext.asset_metadata with final file paths and status.

External Steps (Not part of PipelineOrchestrator's direct loop but integral to the overall process):

• Workspace Preparation and Cleanup: Handled by the code that invokes ProcessingEngine.process() (e.g., main.ProcessingTask, monitor._process_archive_task), typically using utils.workspace_utils. The engine itself creates a sub-temporary directory (engine_temp_dir) within the workspace provided to it by the orchestrator, which it cleans up.
• Prediction and Rule Generation: Also external, performed before ProcessingEngine is called. Generates the SourceRule.
• Optional Blender Script Execution: Triggered externally after successful processing.

This staged pipeline provides a modular and extensible architecture for asset processing, with clear separation of concerns for each step. The AssetProcessingContext ensures that data flows consistently between these stages.r