Metadata reformat done

Documentation/02_Developer_Guide/05_Processing_Pipeline.md

@@ -30,7 +30,7 @@ These stages are executed sequentially once for each asset before the core item
 
 3.  **[`MetadataInitializationStage`](processing/pipeline/stages/metadata_initialization.py:81)** (`processing/pipeline/stages/metadata_initialization.py`):
     *   **Responsibility**: Initializes the `context.asset_metadata` dictionary with base information derived from the [`AssetRule`](rule_structure.py:22), [`SourceRule`](rule_structure.py:40), and [`Configuration`](configuration.py:68). This includes asset name, IDs, source/output paths, timestamps, and initial status.
-    *   **Context Interaction**: Populates `context.asset_metadata` and initializes empty dictionaries for `processed_maps_details` and `merged_maps_details`.
+    *   **Context Interaction**: Populates `context.asset_metadata`. Initializes `context.processed_maps_details` and `context.merged_maps_details` as empty dictionaries (these are used internally by subsequent stages but are not directly part of the final `metadata.json` in their original form).
 
 4.  **[`FileRuleFilterStage`](processing/pipeline/stages/file_rule_filter.py:10)** (`processing/pipeline/stages/file_rule_filter.py`):
     *   **Responsibility**: Filters the [`FileRule`](rule_structure.py:5) objects associated with the asset to determine which individual files should be considered for processing. It identifies and excludes files matching "FILE_IGNORE" rules based on their `item_type`.
@@ -78,11 +78,14 @@ These stages are executed sequentially once for each asset after the core item p
 
 1.  **[`OutputOrganizationStage`](processing/pipeline/stages/output_organization.py:14)** (`processing/pipeline/stages/output_organization.py`):
     *   **Responsibility**: Determines the final output paths for all processed maps (including variants) and extra files based on configured patterns. It copies the temporary files generated by the core stages to these final destinations, creating directories as needed and respecting overwrite settings.
-    *   **Context Interaction**: Reads from `context.processed_maps_details` (using the "MAP_" prefixed `internal_map_type` to get the "standard type" via `get_filename_friendly_map_type` for output naming), `context.files_to_process` (for 'EXTRA' files), `context.output_base_path`, and [`Configuration`](configuration.py:68). Updates entries in `context.processed_maps_details` with final paths and organization status. Populates `context.asset_metadata['final_output_files']`. (Note: Legacy code for `'Processed_With_Variants'` status has been removed from this stage).
+    *   **Context Interaction**: Reads from `context.processed_maps_details`, `context.files_to_process` (for 'EXTRA' files), `context.output_base_path`, and [`Configuration`](configuration.py:68). Updates entries in `context.processed_maps_details` with organization status. Populates `context.asset_metadata['maps']` with the final map structure:
+        *   The `maps` object is a dictionary where keys are standard map types (e.g., "COL", "REFL").
+        *   Each entry contains a `variant_paths` dictionary, where keys are resolution strings (e.g., "8K", "4K") and values are the filenames of the map variants (relative to the asset's output directory).
+        It also populates `context.asset_metadata['final_output_files']` with a list of absolute paths to all generated files (this list itself is not saved in the final `metadata.json`).
 
 2.  **[`MetadataFinalizationAndSaveStage`](processing/pipeline/stages/metadata_finalization_save.py:14)** (`processing/pipeline/stages/metadata_finalization_save.py`):
-    *   **Responsibility**: Finalizes the `context.asset_metadata` (setting end time, final status based on flags). It restructures the processed map details for inclusion, determines the save path for the metadata file based on configuration and patterns, serializes the metadata to JSON, and saves the `metadata.json` file to the final output location.
+    *   **Responsibility**: Finalizes the `context.asset_metadata` (setting final status based on flags). It determines the save path for the metadata file based on configuration and patterns, serializes the `context.asset_metadata` (which now contains the structured `maps` data from `OutputOrganizationStage`) to JSON, and saves the `metadata.json` file.
-    *   **Context Interaction**: Reads from `context.asset_metadata`, `context.processed_maps_details`, `context.merged_maps_details`, `context.output_base_path`, and [`Configuration`](configuration.py:68). Writes the `metadata.json` file and updates `context.asset_metadata` with its final path and status.
+    *   **Context Interaction**: Reads from `context.asset_metadata` (including the `maps` structure), `context.output_base_path`, and [`Configuration`](configuration.py:68). Before saving, it explicitly removes the `final_output_files` key from `context.asset_metadata`. The `processing_end_time` is also no longer added. The `metadata.json` file is written, and `context.asset_metadata` is updated with its final path and status. The older `processed_maps_details` and `merged_maps_details` from the context are not directly included in the JSON.
 
 ## External Steps
 

processing/pipeline/stages/metadata_finalization_save.py

@@ -203,9 +203,8 @@ class MetadataFinalizationAndSaveStage(ProcessingStage):
                     return [make_serializable(i) for i in data]
                 return data
 
-            # final_output_files is populated by OutputOrganizationStage and might be desired.
+            # final_output_files is populated by OutputOrganizationStage. Explicitly remove it as per user request.
-            # The 'maps' structure is now the primary source for map file paths in metadata.
+            context.asset_metadata.pop('final_output_files', None)
-            # context.asset_metadata.pop('final_output_files', None)
             serializable_metadata = make_serializable(context.asset_metadata)
 
             with open(metadata_save_path, 'w') as f:

processing/pipeline/stages/output_organization.py

@@ -116,8 +116,8 @@ class OutputOrganizationStage(ProcessingStage):
                                 # Optionally update variant_detail status if needed
 
                                 # Store relative path in metadata
-                                relative_final_variant_path_str = str(Path(relative_dir_path_str_variant) / Path(output_filename_variant))
+                                # Store only the filename, as it's relative to the metadata.json location
-                                map_metadata_entry['variant_paths'][variant_resolution_key] = relative_final_variant_path_str
+                                map_metadata_entry['variant_paths'][variant_resolution_key] = output_filename_variant
                                 processed_any_variant_successfully = True
 
                         except Exception as e:
@@ -189,7 +189,8 @@ class OutputOrganizationStage(ProcessingStage):
                         # map_type is now the key, so no need to store it inside the entry
                         # map_metadata_entry['map_type'] = base_map_type
                         # Store single path in variant_paths, keyed by its resolution string
-                        map_metadata_entry.setdefault('variant_paths', {})[resolution_str] = str(Path(relative_dir_path_str) / Path(output_filename))
+                        # Store only the filename, as it's relative to the metadata.json location
+                        map_metadata_entry.setdefault('variant_paths', {})[resolution_str] = output_filename
                         # Remove old cleanup logic, as variant_paths is now the standard
                         # if 'variant_paths' in map_metadata_entry:
                         #     del map_metadata_entry['variant_paths']