Removed old prototype
This commit is contained in:
parent
5707867942
commit
74b3d008ea
@ -1,80 +0,0 @@
|
||||
# LLM Asset Classifier Prototype Plan
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Develop a standalone Python prototype module capable of classifying 3D asset files and determining relevant metadata by leveraging a Large Language Model (LLM). This prototype aims to:
|
||||
|
||||
* Handle irregularly named input files where traditional regex/keyword-based presets fail.
|
||||
* Support inputs containing multiple distinct assets within a single source directory or archive.
|
||||
* Provide a flexible foundation for potential future integration into the main Asset Processor Tool.
|
||||
|
||||
## 2. Approach
|
||||
|
||||
We will adopt the following strategies for the prototype development:
|
||||
|
||||
* **Standalone Prototype:** The core LLM interaction and classification logic will be built within a dedicated `llm_prototype/` directory. This isolates development, allows for focused testing, and de-risks the feature before considering integration into the main application.
|
||||
* **Configurable LLM Endpoint:** The prototype will allow users to specify the LLM API endpoint via a configuration file (`config_llm.py`). This enables flexibility in using different LLM providers, including locally hosted models (e.g., via LM Studio) or commercial APIs. API keys will be managed securely via environment variables, not stored in configuration files.
|
||||
* **Multi-Asset Handling via List Output:** To address inputs containing multiple assets (e.g., `Dinesen.zip`, `DG_Imperfections.zip`), the LLM will be prompted to return its findings as a JSON **list**. Each element in the list will represent a single, distinct asset identified within the input file set.
|
||||
* **Chain of Thought (CoT) Prompting:** We will employ a Chain of Thought prompting technique. The LLM will be instructed first to outline its reasoning process (identifying asset groups, assigning files, determining metadata) within `<thinking>` tags before generating the final, structured JSON output. This aims to improve the reliability and accuracy of handling the complex multi-asset identification task.
|
||||
* **Unified Asset Category:** A single `asset_category` field will be used to classify the type of asset. The defined valid categories are: `Model`, `Surface`, `Decal`, `ATLAS`, `Imperfection`.
|
||||
* **JSON Extraction & Validation:** The Python script (`llm_classifier.py`) will parse the full LLM response, extract the JSON list portion following the `<thinking>` block, and perform strict validation against the expected schema and defined category/map type values.
|
||||
* **Minimal Dependencies:** The prototype will aim for minimal external dependencies, primarily using the `requests` library for API communication.
|
||||
|
||||
## 3. Prototype Structure (`llm_prototype/`)
|
||||
|
||||
* `llm_classifier.py`: Main Python script containing the core logic for loading input, formatting prompts, calling the LLM, parsing the response, and validating the output.
|
||||
* `config_llm.py`: Configuration file defining the LLM API endpoint, expected map types, expected asset categories, and the path to the prompt template.
|
||||
* `requirements_llm.txt`: Lists Python dependencies (e.g., `requests`).
|
||||
* `prompt_template.txt`: Text file containing the Chain of Thought prompt template with placeholders for input files and configuration values.
|
||||
* `test_inputs/`: Directory containing example input file lists in JSON format (e.g., `test_inputs/dinesen_example.json`, `test_inputs/imperfections_example.json`).
|
||||
* `README.md`: Instructions detailing setup, configuration (API endpoint, environment variables), and how to run the prototype.
|
||||
|
||||
## 4. Key Schemas
|
||||
|
||||
* **Input to Prototype (Example - `test_inputs/dinesen_example.json`):**
|
||||
```json
|
||||
{
|
||||
"files": [
|
||||
"3-HeartOak-RL-2-5m-300mm_COL-1_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_DISP_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_GLOSS_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_NRM_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_COL-1_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_DISP_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_GLOSS_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_NRM_.jpg"
|
||||
]
|
||||
}
|
||||
```
|
||||
* **Expected LLM Output (Conceptual Structure):**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"asset_name": "3-HeartOak-RL-2-5m-300mm",
|
||||
"asset_category": "Surface", // Unified: Model, Surface, Decal, ATLAS, Imperfection
|
||||
"asset_archetype": "Wood",
|
||||
"file_classifications": [
|
||||
{"input_path": "3-HeartOak-RL-2-5m-300mm_COL-1_.jpg", "classification": "Map", "map_type": "COL"},
|
||||
// ... other HeartOak files
|
||||
]
|
||||
},
|
||||
{
|
||||
"asset_name": "3-Oak-Classic-RL-2-5m-300mm",
|
||||
"asset_category": "Surface",
|
||||
"asset_archetype": "Wood",
|
||||
"file_classifications": [
|
||||
{"input_path": "3-Oak-Classic-RL-2-5m-300mm_COL-1_.jpg", "classification": "Map", "map_type": "COL"},
|
||||
// ... other OakClassic files
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## 5. Next Steps
|
||||
|
||||
1. Create the `llm_prototype/` directory structure. (Already exists, confirmed empty)
|
||||
2. Create the initial placeholder files (`llm_classifier.py`, `config_llm.py`, `requirements_llm.txt`, `prompt_template.txt`, `README.md`).
|
||||
3. Populate `config_llm.py` with initial configuration variables.
|
||||
4. Draft the initial `prompt_template.txt`.
|
||||
5. Create example input files in `test_inputs/`.
|
||||
6. Begin implementing the core logic in `llm_classifier.py`.
|
||||
@ -1,70 +0,0 @@
|
||||
# LLM Asset Classifier Prototype
|
||||
|
||||
This prototype demonstrates using a Large Language Model (LLM) to classify asset files and determine metadata from a list of filenames, particularly for irregularly named inputs and sources containing multiple assets.
|
||||
|
||||
## Setup
|
||||
|
||||
1. **Clone the repository:** If you haven't already, clone the main Asset Processor Tool repository.
|
||||
2. **Navigate to the prototype directory:** `cd llm_prototype/`
|
||||
3. **Create a Python Virtual Environment (Recommended):**
|
||||
```bash
|
||||
python -m venv .venv
|
||||
```
|
||||
4. **Activate the Virtual Environment:**
|
||||
* On Windows: `.venv\Scripts\activate`
|
||||
* On macOS/Linux: `source .venv/bin/activate`
|
||||
5. **Install Dependencies:**
|
||||
```bash
|
||||
pip install -r requirements_llm.txt
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Edit the `config_llm.py` file to configure the LLM API endpoint and other settings.
|
||||
|
||||
* `LLM_API_ENDPOINT`: Set this to the URL of your LLM API. This could be a commercial API (like OpenAI) or a local server (like LM Studio).
|
||||
* `LLM_MODEL_NAME`: If your API requires a specific model name, set it here. Leave empty if not needed.
|
||||
* `LLM_API_KEY_ENV_VAR`: If your API requires an API key, set this to the name of the environment variable where your key is stored.
|
||||
|
||||
**Important:** Do NOT put your API key directly in `config_llm.py`. Store it in an environment variable.
|
||||
|
||||
**Example (Windows PowerShell):**
|
||||
```powershell
|
||||
$env:OPENAI_API_KEY="your_api_key_here"
|
||||
```
|
||||
|
||||
**Example (Linux/macOS Bash):**
|
||||
```bash
|
||||
export OPENAI_API_KEY="your_api_key_here"
|
||||
```
|
||||
*(Replace `OPENAI_API_KEY` and `"your_api_key_here"` as needed based on your `LLM_API_KEY_ENV_VAR` setting and actual key.)*
|
||||
|
||||
## Running the Prototype
|
||||
|
||||
The prototype takes a JSON file containing a list of filenames as input.
|
||||
|
||||
1. **Prepare Input:** Create a JSON file (e.g., in the `test_inputs/` directory) with a structure like this:
|
||||
```json
|
||||
{
|
||||
"files": [
|
||||
"path/to/file1.png",
|
||||
"path/to/model.fbx",
|
||||
"another_file.tif"
|
||||
]
|
||||
}
|
||||
```
|
||||
The paths should be relative paths as they would appear within an extracted asset source directory.
|
||||
|
||||
2. **Execute the script:**
|
||||
```bash
|
||||
python llm_classifier.py <path_to_your_input_json_file>
|
||||
```
|
||||
Replace `<path_to_your_input_json_file>` with the actual path to your input JSON file (e.g., `test_inputs/dinesen_example.json`).
|
||||
|
||||
The script will load the configuration, format the prompt, call the LLM API, extract and validate the JSON response, and print the validated output.
|
||||
|
||||
## Development Notes
|
||||
|
||||
* The LLM prompt template is in `prompt_template.txt`. Modify this file to adjust the instructions given to the LLM.
|
||||
* The core logic is in `llm_classifier.py`. This includes functions for loading config/input/prompt, calling the API, extracting JSON, and validating the output.
|
||||
* The validation logic in `llm_classifier.py` is crucial for ensuring the LLM's output conforms to the expected structure and values.
|
||||
@ -1,94 +0,0 @@
|
||||
# llm_prototype/config_llm.py
|
||||
|
||||
# Configuration for the LLM Asset Classifier Prototype
|
||||
|
||||
# LLM API Endpoint (e.g., for OpenAI, local LLM via LM Studio, etc.)
|
||||
# Example for OpenAI: "https://api.openai.com/v1/chat/completions"
|
||||
# Example for LM Studio: "http://localhost:1234/v1/chat/completions"
|
||||
LLM_API_ENDPOINT = "http://100.65.14.122:1234/v1/chat/completions" # Default to local LM Studio endpoint
|
||||
|
||||
# Optional: LLM Model Name (may be required by some APIs)
|
||||
LLM_MODEL_NAME = "phi-3.5-mini-instruct" # e.g., "gpt-4o-mini", "llama-3-8b-instruct"
|
||||
|
||||
# Environment variable name for the API key (if required by the API)
|
||||
LLM_API_KEY_ENV_VAR = ""
|
||||
|
||||
# Path to the prompt template file (relative to the workspace root)
|
||||
PROMPT_TEMPLATE_PATH = "llm_prototype/prompt_template.txt"
|
||||
|
||||
# Expected internal map types the LLM should classify files into
|
||||
EXPECTED_MAP_TYPES = [
|
||||
"COL", "NRM", "ROUGH", "METAL", "AO", "DISP", "REFL", "MASK", "SSS", "utility"
|
||||
]
|
||||
|
||||
# Examples/synonyms for each map type to guide the LLM
|
||||
MAP_TYPE_EXAMPLES = {
|
||||
"COL": ["Color", "Diffuse", "Albedo"],
|
||||
"NRM": ["Normal"],
|
||||
"ROUGH": ["Roughness"],
|
||||
"METAL": ["Metallic"],
|
||||
"AO": ["Ambient Occlusion"],
|
||||
"DISP": ["Displacement", "Height"],
|
||||
"REFL": ["Reflection"],
|
||||
"MASK": ["Mask", "Alpha", "Opacity"],
|
||||
"SSS": ["Subsurface Scattering"],
|
||||
"utility": ["Utility", "Data", "Helper"]
|
||||
}
|
||||
|
||||
# Expected asset categories the LLM should classify assets into
|
||||
EXPECTED_CATEGORIES = [
|
||||
"Model", "Surface", "Decal", "Atlas", "Imperfection"
|
||||
]
|
||||
|
||||
# Examples/synonyms for each asset category to guide the LLM
|
||||
CATEGORY_EXAMPLES = {
|
||||
"Model": "Represents a asset-set that has a 3D file (e.g., .obj, .fbx, .gltf). Can include accompanying PBR maps as well",
|
||||
"Surface": "Represents a set of textures intended for a material (e.g., PBR texture sets with Color, Normal, Roughness maps).",
|
||||
"Decal": "Represents a texture or set of textures intended to be placed on top of another material (e.g., stickers, grunge overlays). Always has some form of a opacity or mask map",
|
||||
"Atlas": "Represents a texture or set of textures. Often called atlas or trimsheet",
|
||||
"Imperfection": "Represents a single texturemap used to add surface imperfections"
|
||||
}
|
||||
|
||||
# Expected classifications for individual files
|
||||
EXPECTED_CLASSIFICATIONS = [
|
||||
"Map", "Model", "Extra", "Ignored", "Unrecognised"
|
||||
]
|
||||
|
||||
# Placeholder for the input file list in the prompt template
|
||||
FILE_LIST_PLACEHOLDER = "{{FILE_LIST_JSON}}"
|
||||
|
||||
# Placeholder for the expected map types list in the prompt template
|
||||
MAP_TYPES_PLACEHOLDER = "{{EXPECTED_MAP_TYPES}}"
|
||||
|
||||
# Placeholder for the expected categories list in the prompt template
|
||||
CATEGORIES_PLACEHOLDER = "{{EXPECTED_CATEGORIES}}"
|
||||
|
||||
# Placeholder for the category examples in the prompt template
|
||||
CATEGORY_EXAMPLES_PLACEHOLDER = "{{CATEGORY_EXAMPLES}}"
|
||||
|
||||
# Placeholder for the expected classifications list in the prompt template
|
||||
CLASSIFICATIONS_PLACEHOLDER = "{{EXPECTED_CLASSIFICATIONS}}"
|
||||
|
||||
# Placeholder for the expected JSON output schema in the prompt template
|
||||
OUTPUT_SCHEMA_PLACEHOLDER = "{{OUTPUT_SCHEMA}}"
|
||||
|
||||
# The expected JSON output schema structure (used for validation and prompt)
|
||||
# This defines the structure the LLM should return AFTER the <thinking> block
|
||||
OUTPUT_SCHEMA = """
|
||||
[ // Top-level LIST of assets
|
||||
{ // Asset Object
|
||||
"asset_name": "...", // Determined name for the asset set
|
||||
"asset_category": "...", // Must be one of: {{CATEGORIES_PLACEHOLDER}}
|
||||
"asset_archetype": "...", // e.g., Wood, Metal, Fabric, Concrete, Smudge, Scratch, etc.
|
||||
"file_classifications": [ // List of files belonging ONLY to this asset
|
||||
{
|
||||
"input_path": "...", // The original path from the input list
|
||||
"classification": "...", // Must be one of: {{CLASSIFICATIONS_PLACEHOLDER}}
|
||||
"map_type": "..." // Must be one of: {{MAP_TYPES_PLACEHOLDER}}, or null if not a Map
|
||||
}
|
||||
// ... one entry for each file belonging to this asset
|
||||
]
|
||||
}
|
||||
// ... potentially more asset objects if more are detected
|
||||
]
|
||||
"""
|
||||
@ -1,341 +0,0 @@
|
||||
# llm_prototype/llm_classifier.py
|
||||
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import sys
|
||||
import re # Add re import
|
||||
|
||||
# Add the prototype directory to the Python path to import config_llm
|
||||
sys.path.append(os.path.dirname(__file__))
|
||||
import config_llm
|
||||
|
||||
def load_config():
|
||||
"""Loads configuration from config_llm.py."""
|
||||
print("Loading configuration...")
|
||||
return config_llm
|
||||
|
||||
def load_input_files(input_json_path):
|
||||
"""Loads the list of files from an input JSON file."""
|
||||
print(f"Loading input file list from: {input_json_path}")
|
||||
try:
|
||||
with open(input_json_path, 'r') as f:
|
||||
data = json.load(f)
|
||||
if "files" not in data or not isinstance(data["files"], list):
|
||||
raise ValueError("Input JSON must contain a 'files' key with a list of strings.")
|
||||
return data["files"]
|
||||
except FileNotFoundError:
|
||||
print(f"Error: Input file not found at {input_json_path}")
|
||||
sys.exit(1)
|
||||
except json.JSONDecodeError:
|
||||
print(f"Error: Could not decode JSON from {input_json_path}")
|
||||
sys.exit(1)
|
||||
except ValueError as e:
|
||||
print(f"Error in input file format: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
def load_prompt_template(config):
|
||||
"""Loads the prompt template from the specified file."""
|
||||
print(f"Loading prompt template from: {config.PROMPT_TEMPLATE_PATH}")
|
||||
try:
|
||||
with open(config.PROMPT_TEMPLATE_PATH, 'r', encoding='utf-8') as f:
|
||||
return f.read()
|
||||
except FileNotFoundError:
|
||||
print(f"Error: Prompt template file not found at {config.PROMPT_TEMPLATE_PATH}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"Error loading prompt template: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def format_prompt(template, file_list, config):
|
||||
"""Formats the prompt template with dynamic values."""
|
||||
print("Formatting prompt...")
|
||||
file_list_json = json.dumps(file_list, indent=2)
|
||||
|
||||
# Replace placeholders in the template
|
||||
prompt = template.replace(config.FILE_LIST_PLACEHOLDER, file_list_json)
|
||||
prompt = prompt.replace(config.MAP_TYPES_PLACEHOLDER, json.dumps(config.EXPECTED_MAP_TYPES))
|
||||
prompt = prompt.replace(config.CATEGORIES_PLACEHOLDER, json.dumps(config.EXPECTED_CATEGORIES))
|
||||
prompt = prompt.replace(config.CLASSIFICATIONS_PLACEHOLDER, json.dumps(config.EXPECTED_CLASSIFICATIONS))
|
||||
prompt = prompt.replace(config.OUTPUT_SCHEMA_PLACEHOLDER, config.OUTPUT_SCHEMA.strip())
|
||||
|
||||
return prompt
|
||||
|
||||
def call_llm_api(prompt, config):
|
||||
"""
|
||||
Calls the LLM API with the formatted prompt.
|
||||
Handles API key, headers, and basic error checking.
|
||||
"""
|
||||
print(f"Calling LLM API at: {config.LLM_API_ENDPOINT}")
|
||||
|
||||
api_key = os.getenv(config.LLM_API_KEY_ENV_VAR)
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
# Add Authorization header if API key is provided
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
elif "openai.com" in config.LLM_API_ENDPOINT.lower():
|
||||
print(f"Warning: {config.LLM_API_KEY_ENV_VAR} environment variable not set for OpenAI endpoint. API call may fail.")
|
||||
|
||||
|
||||
payload = {
|
||||
"model": config.LLM_MODEL_NAME if config.LLM_MODEL_NAME else "gpt-3.5-turbo", # Use a default if model name is empty
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant that outputs JSON."}, # System message updated
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
# Removed "response_format": {"type": "json_object"} as we expect mixed output (text + JSON)
|
||||
}
|
||||
|
||||
print("\n--- JSON Payload Sent to LLM API ---")
|
||||
print(json.dumps(payload, indent=2))
|
||||
print("------------------------------------\n")
|
||||
|
||||
try:
|
||||
response = requests.post(config.LLM_API_ENDPOINT, headers=headers, json=payload)
|
||||
response.raise_for_status() # Raise an HTTPError for bad responses (4xx or 5xx)
|
||||
|
||||
# Check if the response is directly a JSON object (as requested by response_format)
|
||||
# Some APIs might return the JSON directly if response_format is supported and successful.
|
||||
try:
|
||||
json_response = response.json()
|
||||
# Check if the JSON response looks like a chat completion response
|
||||
if 'choices' in json_response and len(json_response['choices']) > 0:
|
||||
return json_response # It's a standard chat completion response
|
||||
else:
|
||||
# It might be the raw JSON output if the API returned it directly
|
||||
print("Received direct JSON response (not standard chat completion format).")
|
||||
# Wrap it in a structure similar to chat completion for consistent processing
|
||||
return {"choices": [{"message": {"content": json.dumps(json_response)}}]}
|
||||
except json.JSONDecodeError:
|
||||
# If it's not JSON, assume it's plain text content
|
||||
print("Received non-JSON response. Treating as plain text.")
|
||||
return {"choices": [{"message": {"content": response.text}}]}
|
||||
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"Error calling LLM API: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
def extract_json_from_response(response_data):
|
||||
"""
|
||||
Extracts the main JSON object or list from the LLM's response content.
|
||||
It handles markdown fences, reasoning tags (e.g., <think>), and aims
|
||||
to find the first complete JSON structure ({...} or [...]).
|
||||
"""
|
||||
print("Extracting JSON from LLM response...")
|
||||
|
||||
# Look for the content of the first message from the assistant
|
||||
assistant_message_content = ""
|
||||
if 'choices' in response_data and len(response_data['choices']) > 0:
|
||||
message = response_data['choices'][0].get('message', {})
|
||||
assistant_message_content = message.get('content', '')
|
||||
|
||||
if not assistant_message_content:
|
||||
print("Warning: LLM response content is empty or not found in expected structure.")
|
||||
print(f"Full response data: {response_data}")
|
||||
return [] # Return empty list if no content
|
||||
|
||||
content = assistant_message_content.strip()
|
||||
|
||||
# 1. Strip markdown code fences (```json ... ``` or ``` ... ```)
|
||||
content = re.sub(r'^```(?:json)?\s*', '', content, flags=re.IGNORECASE)
|
||||
content = re.sub(r'\s*```$', '', content)
|
||||
content = content.strip()
|
||||
|
||||
print("\n--- Content After Stripping Fences ---")
|
||||
print(content)
|
||||
print("---------------------------------------\n")
|
||||
|
||||
# 2. Remove reasoning tags like <think>...</think> (non-greedy)
|
||||
# Consider making tag removal more general if other tags appear
|
||||
print("\n--- Content BEFORE Removing <think> Tags ---")
|
||||
print(repr(content)) # Using repr() to see hidden characters like newlines
|
||||
content = re.sub(r'<think>.*?</think>', '', content, flags=re.DOTALL | re.IGNORECASE)
|
||||
print("\n--- Content AFTER Removing <think> Tags ---") # Added this line
|
||||
print(repr(content)) # Using repr() to see hidden characters like newlines
|
||||
content = content.strip()
|
||||
|
||||
# Original print statement, now correctly indented and showing the final cleaned content before JSON parsing attempt
|
||||
print("\n--- Final Content Before JSON Parsing Attempt ---")
|
||||
print(content)
|
||||
print("-------------------------------------------------\n")
|
||||
|
||||
if not content:
|
||||
print("Error: LLM response content is empty after stripping fences and tags.")
|
||||
return [] # Return empty list if nothing remains
|
||||
|
||||
# 3. Find the first opening bracket or brace indicating start of JSON
|
||||
first_bracket_index = content.find('[')
|
||||
first_brace_index = content.find('{')
|
||||
|
||||
start_index = -1
|
||||
if first_bracket_index != -1 and first_brace_index != -1:
|
||||
start_index = min(first_bracket_index, first_brace_index)
|
||||
elif first_bracket_index != -1:
|
||||
start_index = first_bracket_index
|
||||
elif first_brace_index != -1:
|
||||
start_index = first_brace_index
|
||||
|
||||
if start_index == -1:
|
||||
print("Error: Could not find starting '[' or '{' in the processed content.")
|
||||
print(f"Processed content snippet: {content[:500]}...")
|
||||
return [] # Return empty list if no JSON start found
|
||||
|
||||
# 4. Attempt to find the corresponding closing bracket/brace and parse
|
||||
# This uses json.JSONDecoder.raw_decode to find the first valid JSON object/array.
|
||||
potential_json_str = content[start_index:]
|
||||
json_decoder = json.JSONDecoder()
|
||||
try:
|
||||
# Use decode with raw_decode to find the first valid JSON object/array
|
||||
# and its end position in the string.
|
||||
parsed_json, end_pos = json_decoder.raw_decode(potential_json_str)
|
||||
print(f"Successfully parsed JSON ending at index {start_index + end_pos}.")
|
||||
# Optional: Log the extracted part: print(f"Extracted JSON string: {potential_json_str[:end_pos]}")
|
||||
return parsed_json
|
||||
except json.JSONDecodeError as e:
|
||||
# This error means no valid JSON object was found starting at start_index
|
||||
print(f"Error: Could not decode JSON starting from index {start_index}: {e}")
|
||||
print(f"Content snippet starting at index {start_index}: {potential_json_str[:500]}...")
|
||||
|
||||
# Fallback: Try the original naive approach (first '['/'{' to last ']'/'}')
|
||||
# This might capture the JSON if it's the last element, even with preceding noise
|
||||
# that confused raw_decode.
|
||||
print("Attempting fallback: finding last ']' or '}'...")
|
||||
last_bracket_index = content.rfind(']')
|
||||
last_brace_index = content.rfind('}')
|
||||
end_index = max(last_bracket_index, last_brace_index)
|
||||
|
||||
if end_index > start_index:
|
||||
fallback_json_string = content[start_index : end_index + 1]
|
||||
print(f"Fallback attempting to parse: {fallback_json_string[:500]}...")
|
||||
try:
|
||||
parsed_json = json.loads(fallback_json_string)
|
||||
print("Successfully parsed JSON using fallback method.")
|
||||
return parsed_json
|
||||
except json.JSONDecodeError as fallback_e:
|
||||
print(f"Fallback JSON parsing also failed: {fallback_e}")
|
||||
return [] # Return empty list if both methods fail
|
||||
else:
|
||||
print("Fallback failed: Could not find suitable closing bracket/brace.")
|
||||
return [] # Return empty list if fallback indices are invalid
|
||||
|
||||
|
||||
|
||||
|
||||
def validate_llm_output(llm_output, config):
|
||||
"""Validates the structure and content of the parsed LLM output JSON."""
|
||||
print("Validating LLM output...")
|
||||
|
||||
if not isinstance(llm_output, list):
|
||||
print("Validation Error: LLM output is not a JSON list.")
|
||||
return False
|
||||
|
||||
if not llm_output:
|
||||
print("Validation Warning: LLM output list is empty.")
|
||||
# Depending on requirements, an empty list might be valid if no assets were found.
|
||||
# For now, we'll allow it but warn.
|
||||
|
||||
required_asset_keys = ["asset_name", "asset_category", "asset_archetype", "file_classifications"]
|
||||
required_file_keys = ["input_path", "classification", "map_type"]
|
||||
|
||||
for i, asset in enumerate(llm_output):
|
||||
if not isinstance(asset, dict):
|
||||
print(f"Validation Error: Asset item at index {i} is not a dictionary.")
|
||||
return False
|
||||
|
||||
# Validate required asset keys
|
||||
for key in required_asset_keys:
|
||||
if key not in asset:
|
||||
print(f"Validation Error: Asset item at index {i} is missing required key: '{key}'.")
|
||||
return False
|
||||
|
||||
# Validate asset_category value
|
||||
if asset["asset_category"] not in config.EXPECTED_CATEGORIES:
|
||||
print(f"Validation Error: Asset item at index {i} has invalid asset_category: '{asset['asset_category']}'. Expected one of: {config.EXPECTED_CATEGORIES}")
|
||||
return False
|
||||
|
||||
# Validate file_classifications list
|
||||
if not isinstance(asset["file_classifications"], list):
|
||||
print(f"Validation Error: 'file_classifications' for asset at index {i} is not a list.")
|
||||
return False
|
||||
|
||||
for j, file_info in enumerate(asset["file_classifications"]):
|
||||
if not isinstance(file_info, dict):
|
||||
print(f"Validation Error: File classification item at asset index {i}, file index {j} is not a dictionary.")
|
||||
return False
|
||||
|
||||
# Validate required file keys
|
||||
for key in required_file_keys:
|
||||
if key not in file_info:
|
||||
print(f"Validation Error: File classification item at asset index {i}, file index {j} is missing required key: '{key}'.")
|
||||
return False
|
||||
|
||||
# Validate classification value
|
||||
if file_info["classification"] not in config.EXPECTED_CLASSIFICATIONS:
|
||||
print(f"Validation Error: File classification item at asset index {i}, file index {j} has invalid classification: '{file_info['classification']}'. Expected one of: {config.EXPECTED_CLASSIFICATIONS}")
|
||||
return False
|
||||
|
||||
# Validate map_type value if classification is "Map"
|
||||
if file_info["classification"] == "Map":
|
||||
if file_info["map_type"] is None or file_info["map_type"] not in config.EXPECTED_MAP_TYPES:
|
||||
print(f"Validation Error: File classification item at asset index {i}, file index {j} is classified as 'Map' but has invalid or missing map_type: '{file_info['map_type']}'. Expected one of: {config.EXPECTED_MAP_TYPES}")
|
||||
return False
|
||||
elif file_info["map_type"] is not None:
|
||||
print(f"Validation Warning: File classification item at asset index {i}, file index {j} is not classified as 'Map' but has a map_type: '{file_info['map_type']}'. map_type should be null.")
|
||||
# This is a warning, not a strict error, as some LLMs might include it.
|
||||
|
||||
# Validate input_path is a string
|
||||
if not isinstance(file_info["input_path"], str) or not file_info["input_path"]:
|
||||
print(f"Validation Error: File classification item at asset index {i}, file index {j} has invalid or empty input_path: '{file_info['input_path']}'.")
|
||||
return False
|
||||
|
||||
|
||||
print("LLM output validation successful.")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function to run the LLM classifier prototype."""
|
||||
if len(sys.argv) != 2:
|
||||
print("Usage: python llm_classifier.py <path_to_input_json>")
|
||||
sys.exit(1)
|
||||
|
||||
input_json_path = sys.argv[1]
|
||||
|
||||
config = load_config()
|
||||
file_list = load_input_files(input_json_path)
|
||||
prompt_template = load_prompt_template(config)
|
||||
prompt = format_prompt(prompt_template, file_list, config)
|
||||
|
||||
# print("\n--- Generated Prompt ---")
|
||||
# print(prompt)
|
||||
# print("------------------------\n")
|
||||
|
||||
llm_response_data = call_llm_api(prompt, config)
|
||||
|
||||
# print("\n--- Raw LLM Response ---")
|
||||
# print(json.dumps(llm_response_data, indent=2))
|
||||
# print("------------------------\n")
|
||||
|
||||
llm_output_json = extract_json_from_response(llm_response_data)
|
||||
|
||||
# Add check for empty extracted JSON
|
||||
if not llm_output_json:
|
||||
print("\nError: Extracted JSON list is empty. LLM response or extraction failed.")
|
||||
sys.exit(1)
|
||||
|
||||
if validate_llm_output(llm_output_json, config):
|
||||
print("\n--- Validated LLM Output JSON ---")
|
||||
print(json.dumps(llm_output_json, indent=2))
|
||||
print("---------------------------------\n")
|
||||
print("Prototype run complete. LLM output is valid.")
|
||||
else:
|
||||
print("\nPrototype run failed due to LLM output validation errors.")
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,98 +0,0 @@
|
||||
# llm_prototype/prompt_template.txt
|
||||
|
||||
You are an expert 3D asset file classifier. Your task is to analyze a list of filenames from a single asset source (which may contain multiple distinct assets) and provide a structured JSON output detailing the identified assets and their file classifications.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
1. Analyze the provided list of filenames.
|
||||
2. Identify distinct asset groups within the list based on naming patterns, file types, or other logical groupings.
|
||||
3. For each identified asset group, determine:
|
||||
* A concise `asset_name`.
|
||||
* The `asset_category`. This must be one of the following exact strings: {{CATEGORIES_PLACEHOLDER}}. Refer to the descriptions below for guidance: {{CATEGORY_EXAMPLES}}. Note: The 'Imperfection' category typically contains a single map file.
|
||||
* The `asset_archetype` (e.g., Wood, Metal, Fabric, Concrete, Smudge, Scratch, etc.).
|
||||
4. For each file in the original input list, determine which asset group it belongs to (if any).
|
||||
4.5. If a file does not belong to a distinct asset group (e.g., preview images, notes), classify it accordingly ("Extra", "Ignored", "Unrecognised") and include it in the "file_classifications" list of the *first* asset object in your output.
|
||||
5. For each file, determine its `classification`. This must be one of the following exact strings: Map, Model, Extra, Ignored, Unrecognised. (Using placeholder: {{CLASSIFICATIONS_PLACEHOLDER}})
|
||||
6. If the classification is "Map", determine the `map_type`. This must be one of the following exact strings: {{EXPECTED_MAP_TYPES}}. If the classification is not "Map", the `map_type` should be `null`. Use the most appropriate map type from the list. For imperfection maps, the map_type should typically be "utility". Common synonyms for these map types include: COL (Color, Diffuse, Albedo), NRM (Normal), ROUGH (Roughness), METAL (Metallic), AO (Ambient Occlusion), DISP (Displacement, Height), REFL (Reflection), MASK (Mask, Alpha, Opacity), SSS (Subsurface Scattering), utility (Utility, Data, Helper).
|
||||
7. Output your reasoning process within <thinking> and </thinking> tags. Explain how you identified the asset groups and classified the files.
|
||||
8. After the </thinking> tag, output ONLY the final JSON list structure based on your analysis. **Ensure the JSON is complete and valid.** Do not include any other text outside the JSON.
|
||||
**STRICT INSTRUCTION: Output ONLY the final JSON list structure after the </thinking> tag. The output MUST be a valid JSON list `[...]` containing one object `{...}` for EACH distinct asset identified. EACH asset object MUST include the keys "asset_name", "asset_category", "asset_archetype", and "file_classifications". ABSOLUTELY NO markdown code fences (```json) or comments (`//` or `#`) are allowed in the JSON output.**
|
||||
|
||||
Input File List:
|
||||
{{FILE_LIST_JSON}}
|
||||
|
||||
Category Examples:
|
||||
{{CATEGORY_EXAMPLES}}
|
||||
|
||||
## Examples
|
||||
|
||||
Here are a few examples of input file lists and the EXACT expected JSON output format.
|
||||
|
||||
**Example 1: Single PBR Surface**
|
||||
Input:
|
||||
[
|
||||
"Wood_Plank_01_COL_1K.jpg",
|
||||
"Wood_Plank_01_NRM_1K.jpg",
|
||||
"Wood_Plank_01_ROUGH_1K.jpg"
|
||||
]
|
||||
Expected Output:
|
||||
[
|
||||
{
|
||||
"asset_name": "Wood_Plank_01",
|
||||
"asset_category": "Surface",
|
||||
"asset_archetype": "Wood",
|
||||
"file_classifications": [
|
||||
{"input_path": "Wood_Plank_01_COL_1K.jpg", "classification": "Map", "map_type": "COL"},
|
||||
{"input_path": "Wood_Plank_01_NRM_1K.jpg", "classification": "Map", "map_type": "NRM"},
|
||||
{"input_path": "Wood_Plank_01_ROUGH_1K.jpg", "classification": "Map", "map_type": "ROUGH"}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
**Example 2: Multiple Imperfection Maps**
|
||||
Input:
|
||||
[
|
||||
"DG-01-Smudges.tif",
|
||||
"DG-02-Scratches.tif",
|
||||
"DG-03-Smudges-Scratches.tif",
|
||||
"preview.jpg",
|
||||
"notes.txt"
|
||||
]
|
||||
Expected Output:
|
||||
[
|
||||
{
|
||||
"asset_name": "Smudges",
|
||||
"asset_category": "Imperfection",
|
||||
"asset_archetype": "Smudge",
|
||||
"file_classifications": [
|
||||
{"input_path": "DG-01-Smudges.tif", "classification": "Map", "map_type": "utility"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"asset_name": "Scratches",
|
||||
"asset_category": "Imperfection",
|
||||
"asset_archetype": "Scratch",
|
||||
"file_classifications": [
|
||||
{"input_path": "DG-02-Scratches.tif", "classification": "Map", "map_type": "utility"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"asset_name": "Smudges-Scratches",
|
||||
"asset_category": "Imperfection",
|
||||
"asset_archetype": "Mixed",
|
||||
"file_classifications": [
|
||||
{"input_path": "DG-03-Smudges-Scratches.tif", "classification": "Map", "map_type": "utility"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"asset_name": "Extra Files",
|
||||
"asset_category": "Extra",
|
||||
"asset_archetype": null,
|
||||
"file_classifications": [
|
||||
{"input_path": "preview.jpg", "classification": "Extra", "map_type": null},
|
||||
{"input_path": "notes.txt", "classification": "Ignored", "map_type": null}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
Input File List:
|
||||
@ -1,2 +0,0 @@
|
||||
# llm_prototype/requirements_llm.txt
|
||||
requests
|
||||
@ -1,12 +0,0 @@
|
||||
{
|
||||
"files": [
|
||||
"3-HeartOak-RL-2-5m-300mm_COL-1_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_DISP_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_GLOSS_.jpg",
|
||||
"3-HeartOak-RL-2-5m-300mm_NRM_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_COL-1_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_DISP_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_GLOSS_.jpg",
|
||||
"3-Oak-Classic-RL-2-5m-300mm_NRM_.jpg"
|
||||
]
|
||||
}
|
||||
@ -1,9 +0,0 @@
|
||||
{
|
||||
"files": [
|
||||
"DG-01-Smudges.tif",
|
||||
"DG-02-Scratches.tif",
|
||||
"DG-03-Smudges-Scratches.tif",
|
||||
"preview.jpg",
|
||||
"notes.txt"
|
||||
]
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user