PlantGuide/Docs/Phase2_plan.md

Phase 2: On-Device ML

Goal: Offline plant identification with Core ML

Prerequisites: Phase 1 complete (camera capture working, folder structure in place)

---

Tasks

2.1 Download PlantNet-300K Pre-trained Weights

• Create scripts/ directory at project root
• Download ResNet50 weights from Zenodo:

  wget https://zenodo.org/records/4726653/files/resnet50_weights_best_acc.tar
  

• Verify file integrity (expected ~100MB)
• Document download source and version in scripts/README.md

Acceptance Criteria: resnet50_weights_best_acc.tar file present and verified

---

2.2 Convert Model to Core ML

• Install Python dependencies:

  pip install torch torchvision coremltools pillow numpy
  

• Create scripts/convert_plantnet_to_coreml.py:
  • Load ResNet50 architecture with 1,081 output classes
  • Load PlantNet-300K weights
  • Trace model with dummy input
  • Configure image input preprocessing (RGB, 224x224, normalized)
  • Convert to ML Program format
  • Target iOS 17+ deployment
  • Use FLOAT16 precision for performance
• Run conversion script
• Verify output PlantNet300K.mlpackage created successfully

Acceptance Criteria: PlantNet300K.mlpackage generated without errors, file size ~50-100MB

---

2.3 Add Core ML Model to Xcode

• Copy PlantNet300K.mlpackage to ML/Models/
• Add to Xcode project target
• Verify Xcode generates PlantNet300K.swift interface
• Configure model to compile at build time (not runtime)
• Test that project still builds successfully

Acceptance Criteria: Model visible in Xcode, auto-generated Swift interface available

---

2.4 Create Plant Labels JSON

• Download species list from PlantNet-300K dataset
• Create Resources/PlantLabels.json with structure:

  {
    "labels": [
      {
        "index": 0,
        "scientificName": "Acer campestre",
        "commonNames": ["Field Maple", "Hedge Maple"],
        "family": "Sapindaceae"
      }
    ],
    "version": "1.0",
    "source": "PlantNet-300K"
  }
  

• Ensure all 1,081 species mapped correctly
• Validate JSON syntax
• Create ML/Services/PlantLabelService.swift to load and query labels

Acceptance Criteria: JSON contains 1,081 species entries, loads without parsing errors

---

2.5 Implement Plant Classification Service

• Create ML/Services/PlantClassificationService.swift
• Define PlantClassificationServiceProtocol:

  protocol PlantClassificationServiceProtocol: Sendable {
      func classify(image: CGImage) async throws -> [PlantPrediction]
  }
  

• Create PlantPrediction struct:
  • speciesIndex: Int
  • confidence: Float
  • scientificName: String
  • commonNames: [String]
• Implement using Vision framework:
  • Create VNCoreMLRequest with PlantNet300K model
  • Configure imageCropAndScaleOption to .centerCrop
  • Process results from VNClassificationObservation
• Return top 10 predictions sorted by confidence
• Handle model loading errors gracefully

Acceptance Criteria: Service returns predictions for any valid CGImage input

---

2.6 Create Image Preprocessor

• Create ML/Preprocessing/ImagePreprocessor.swift
• Define ImagePreprocessorProtocol:

  protocol ImagePreprocessorProtocol: Sendable {
      func prepare(image: UIImage) -> CGImage?
      func prepare(data: Data) -> CGImage?
  }
  

• Implement preprocessing pipeline:
  • Convert UIImage to CGImage
  • Handle orientation correction (EXIF)
  • Resize to 224x224 if needed (Vision handles this, but validate)
  • Convert color space to sRGB if needed
• Add validation for minimum image dimensions
• Handle edge cases (nil image, corrupt data)

Acceptance Criteria: Preprocessor handles images from camera and photo library correctly

---

2.7 Build Identify Plant On-Device Use Case

• Create Domain/UseCases/Identification/IdentifyPlantOnDeviceUseCase.swift
• Define use case protocol:

  protocol IdentifyPlantOnDeviceUseCaseProtocol: Sendable {
      func execute(image: UIImage) async throws -> PlantIdentification
  }
  

• Implement use case:
  • Preprocess image
  • Call classification service
  • Map predictions to PlantIdentification entity
  • Set source to .onDevice
  • Record timestamp
• Add to DIContainer factory methods
• Create unit test with mock classification service

Acceptance Criteria: Use case integrates preprocessor and classifier, returns domain entity

---

2.8 Create Identification View

• Create Presentation/Scenes/Identification/IdentificationView.swift
• Create Presentation/Scenes/Identification/IdentificationViewModel.swift
• Implement UI states:
  • Loading (during inference)
  • Success (show results)
  • Error (show retry option)
• Display captured image at top
• Show list of top 10 species matches
• Add "Identify Again" button
• Add "Save to Collection" button (disabled for Phase 2)
• Navigate from CameraView after capture

Acceptance Criteria: View displays results after photo capture, handles all states

---

2.9 Build Species Match Components

• Create Presentation/Common/Components/SpeciesMatchCard.swift:
  • Scientific name (primary text)
  • Common names (secondary text)
  • Confidence score
  • Ranking number (1-10)
  • Chevron for detail navigation (future)
• Create Presentation/Common/Components/ConfidenceIndicator.swift:
  • Visual progress bar
  • Percentage text
  • Color coding:
    • Green: > 70%
    • Yellow: 40-70%
    • Red: < 40%
• Style components with consistent design language
• Ensure accessibility labels set correctly

Acceptance Criteria: Components render correctly with sample data, accessible

---

2.10 Performance Testing

• Create ML/Tests/ClassificationPerformanceTests.swift
• Measure inference time on real device:
  • Test with 10 different plant images
  • Record min/max/average times
  • Target: < 500ms average
• Measure memory usage during inference:
  • Target: < 200MB peak
• Test on oldest supported device (if available)
• Profile with Instruments (Core ML template)
• Document results in Docs/Phase2_performance_results.md

Acceptance Criteria: Average inference < 500ms on target device

---

End-of-Phase Validation

Functional Verification

Test	Steps	Expected Result	Status
Model Loads	Launch app	No model loading errors in console	[ ]
Labels Load	Launch app	PlantLabels.json parsed successfully	[ ]
Camera → Identify	Take photo, wait	Identification results appear	[ ]
Results Display	View results	10 species shown with confidence scores	[ ]
Confidence Colors	View varied results	Colors match confidence levels	[ ]
Loading State	Take photo	Loading indicator shown during inference	[ ]
Error Handling	Force error (mock)	Error view displays with retry	[ ]
Retry Flow	Tap retry	Returns to camera	[ ]

Code Quality Verification

Check	Criteria	Status
Build	Project builds with zero warnings	[ ]
Architecture	ML code isolated in ML/ folder	[ ]
Protocols	Classification service uses protocol	[ ]
Sendable	All ML services are Sendable	[ ]
Use Case	Identification logic in use case, not ViewModel	[ ]
DI Container	Classification service injected via container	[ ]
Error Types	Custom errors defined for ML failures	[ ]
Unit Tests	Use case has at least one unit test	[ ]

Performance Verification

Metric	Target	Actual	Status
Model Load Time	< 2 seconds		[ ]
Inference Time (avg)	< 500ms		[ ]
Inference Time (max)	< 1000ms		[ ]
Memory During Inference	< 200MB		[ ]
Memory After Inference	Returns to baseline		[ ]
App Size Increase	< 100MB (model)		[ ]

Accuracy Verification

Test Image	Expected Top Match	Actual Top Match	Confidence	Status
Rose photo	Rosa sp.			[ ]
Oak leaf	Quercus sp.			[ ]
Sunflower	Helianthus annuus			[ ]
Tulip	Tulipa sp.			[ ]
Fern	Pteridophyta sp.			[ ]

---

Phase 2 Completion Checklist

• All 10 tasks completed
• All functional tests pass
• All code quality checks pass
• All performance targets met
• Accuracy spot-checked with 5+ real plant images
• Core ML model included in app bundle
• PlantLabels.json contains all 1,081 species
• Performance results documented
• Code committed with descriptive message
• Ready for Phase 3 (PlantNet API Integration)

---

Error Handling

ML-Specific Errors

enum PlantClassificationError: Error, LocalizedError {
    case modelLoadFailed
    case imagePreprocessingFailed
    case inferenceTimeout
    case noResultsReturned
    case labelsNotFound

    var errorDescription: String? {
        switch self {
        case .modelLoadFailed:
            return "Unable to load plant identification model"
        case .imagePreprocessingFailed:
            return "Unable to process image for analysis"
        case .inferenceTimeout:
            return "Identification took too long"
        case .noResultsReturned:
            return "No plant species identified"
        case .labelsNotFound:
            return "Plant database not available"
        }
    }
}

---

Notes

• Vision framework handles image scaling/cropping automatically via imageCropAndScaleOption
• Core ML models should be loaded once and reused (expensive to initialize)
• Use MLModelConfiguration to control compute units (CPU, GPU, Neural Engine)
• FLOAT16 precision reduces model size with minimal accuracy loss
• Test on real device early - simulator performance not representative
• Consider background thread for inference to keep UI responsive
• PlantNet-300K trained on European flora - accuracy varies by region

---

Dependencies

Dependency	Type	Notes
PlantNet300K.mlpackage	Core ML Model	~50-100MB, bundled
PlantLabels.json	Data File	1,081 species, bundled
Vision.framework	System	iOS 11+
CoreML.framework	System	iOS 11+

---

Risk Mitigation

Risk	Mitigation
Model too large for App Store	Use on-demand resources or model quantization
Inference too slow	Profile with Instruments, use Neural Engine
Low accuracy	Phase 3 adds API fallback for confirmation
Memory pressure	Unload model when not in use (trade-off: reload time)
Unsupported species	Show "Unknown" with low confidence, suggest API