PlantGuide/Docs/Phase2_plan.md

# Phase 2: On-Device ML

**Goal:** Offline plant identification with Core ML

**Prerequisites:** Phase 1 complete (camera capture working, folder structure in place)

---

## Tasks

### 2.1 Download PlantNet-300K Pre-trained Weights
- [ ] Create `scripts/` directory at project root
- [ ] Download ResNet50 weights from Zenodo:
  ```bash
  wget https://zenodo.org/records/4726653/files/resnet50_weights_best_acc.tar
  ```
- [ ] Verify file integrity (expected ~100MB)
- [ ] Document download source and version in `scripts/README.md`

**Acceptance Criteria:** `resnet50_weights_best_acc.tar` file present and verified

---

### 2.2 Convert Model to Core ML
- [ ] Install Python dependencies:
  ```bash
  pip install torch torchvision coremltools pillow numpy
  ```
- [ ] Create `scripts/convert_plantnet_to_coreml.py`:
  - Load ResNet50 architecture with 1,081 output classes
  - Load PlantNet-300K weights
  - Trace model with dummy input
  - Configure image input preprocessing (RGB, 224x224, normalized)
  - Convert to ML Program format
  - Target iOS 17+ deployment
  - Use FLOAT16 precision for performance
- [ ] Run conversion script
- [ ] Verify output `PlantNet300K.mlpackage` created successfully

**Acceptance Criteria:** `PlantNet300K.mlpackage` generated without errors, file size ~50-100MB

---

### 2.3 Add Core ML Model to Xcode
- [ ] Copy `PlantNet300K.mlpackage` to `ML/Models/`
- [ ] Add to Xcode project target
- [ ] Verify Xcode generates `PlantNet300K.swift` interface
- [ ] Configure model to compile at build time (not runtime)
- [ ] Test that project still builds successfully

**Acceptance Criteria:** Model visible in Xcode, auto-generated Swift interface available

---

### 2.4 Create Plant Labels JSON
- [ ] Download species list from PlantNet-300K dataset
- [ ] Create `Resources/PlantLabels.json` with structure:
  ```json
  {
    "labels": [
      {
        "index": 0,
        "scientificName": "Acer campestre",
        "commonNames": ["Field Maple", "Hedge Maple"],
        "family": "Sapindaceae"
      }
    ],
    "version": "1.0",
    "source": "PlantNet-300K"
  }
  ```
- [ ] Ensure all 1,081 species mapped correctly
- [ ] Validate JSON syntax
- [ ] Create `ML/Services/PlantLabelService.swift` to load and query labels

**Acceptance Criteria:** JSON contains 1,081 species entries, loads without parsing errors

---

### 2.5 Implement Plant Classification Service
- [ ] Create `ML/Services/PlantClassificationService.swift`
- [ ] Define `PlantClassificationServiceProtocol`:
  ```swift
  protocol PlantClassificationServiceProtocol: Sendable {
      func classify(image: CGImage) async throws -> [PlantPrediction]
  }
  ```
- [ ] Create `PlantPrediction` struct:
  - `speciesIndex: Int`
  - `confidence: Float`
  - `scientificName: String`
  - `commonNames: [String]`
- [ ] Implement using Vision framework:
  - Create `VNCoreMLRequest` with PlantNet300K model
  - Configure `imageCropAndScaleOption` to `.centerCrop`
  - Process results from `VNClassificationObservation`
- [ ] Return top 10 predictions sorted by confidence
- [ ] Handle model loading errors gracefully

**Acceptance Criteria:** Service returns predictions for any valid CGImage input

---

### 2.6 Create Image Preprocessor
- [ ] Create `ML/Preprocessing/ImagePreprocessor.swift`
- [ ] Define `ImagePreprocessorProtocol`:
  ```swift
  protocol ImagePreprocessorProtocol: Sendable {
      func prepare(image: UIImage) -> CGImage?
      func prepare(data: Data) -> CGImage?
  }
  ```
- [ ] Implement preprocessing pipeline:
  - Convert UIImage to CGImage
  - Handle orientation correction (EXIF)
  - Resize to 224x224 if needed (Vision handles this, but validate)
  - Convert color space to sRGB if needed
- [ ] Add validation for minimum image dimensions
- [ ] Handle edge cases (nil image, corrupt data)

**Acceptance Criteria:** Preprocessor handles images from camera and photo library correctly

---

### 2.7 Build Identify Plant On-Device Use Case
- [ ] Create `Domain/UseCases/Identification/IdentifyPlantOnDeviceUseCase.swift`
- [ ] Define use case protocol:
  ```swift
  protocol IdentifyPlantOnDeviceUseCaseProtocol: Sendable {
      func execute(image: UIImage) async throws -> PlantIdentification
  }
  ```
- [ ] Implement use case:
  - Preprocess image
  - Call classification service
  - Map predictions to `PlantIdentification` entity
  - Set `source` to `.onDevice`
  - Record timestamp
- [ ] Add to DIContainer factory methods
- [ ] Create unit test with mock classification service

**Acceptance Criteria:** Use case integrates preprocessor and classifier, returns domain entity

---

### 2.8 Create Identification View
- [ ] Create `Presentation/Scenes/Identification/IdentificationView.swift`
- [ ] Create `Presentation/Scenes/Identification/IdentificationViewModel.swift`
- [ ] Implement UI states:
  - Loading (during inference)
  - Success (show results)
  - Error (show retry option)
- [ ] Display captured image at top
- [ ] Show list of top 10 species matches
- [ ] Add "Identify Again" button
- [ ] Add "Save to Collection" button (disabled for Phase 2)
- [ ] Navigate from CameraView after capture

**Acceptance Criteria:** View displays results after photo capture, handles all states

---

### 2.9 Build Species Match Components
- [ ] Create `Presentation/Common/Components/SpeciesMatchCard.swift`:
  - Scientific name (primary text)
  - Common names (secondary text)
  - Confidence score
  - Ranking number (1-10)
  - Chevron for detail navigation (future)
- [ ] Create `Presentation/Common/Components/ConfidenceIndicator.swift`:
  - Visual progress bar
  - Percentage text
  - Color coding:
    - Green: > 70%
    - Yellow: 40-70%
    - Red: < 40%
- [ ] Style components with consistent design language
- [ ] Ensure accessibility labels set correctly

**Acceptance Criteria:** Components render correctly with sample data, accessible

---

### 2.10 Performance Testing
- [ ] Create `ML/Tests/ClassificationPerformanceTests.swift`
- [ ] Measure inference time on real device:
  - Test with 10 different plant images
  - Record min/max/average times
  - Target: < 500ms average
- [ ] Measure memory usage during inference:
  - Target: < 200MB peak
- [ ] Test on oldest supported device (if available)
- [ ] Profile with Instruments (Core ML template)
- [ ] Document results in `Docs/Phase2_performance_results.md`

**Acceptance Criteria:** Average inference < 500ms on target device

---

## End-of-Phase Validation

### Functional Verification

| Test | Steps | Expected Result | Status |
|------|-------|-----------------|--------|
| Model Loads | Launch app | No model loading errors in console | [ ] |
| Labels Load | Launch app | PlantLabels.json parsed successfully | [ ] |
| Camera → Identify | Take photo, wait | Identification results appear | [ ] |
| Results Display | View results | 10 species shown with confidence scores | [ ] |
| Confidence Colors | View varied results | Colors match confidence levels | [ ] |
| Loading State | Take photo | Loading indicator shown during inference | [ ] |
| Error Handling | Force error (mock) | Error view displays with retry | [ ] |
| Retry Flow | Tap retry | Returns to camera | [ ] |

### Code Quality Verification

| Check | Criteria | Status |
|-------|----------|--------|
| Build | Project builds with zero warnings | [ ] |
| Architecture | ML code isolated in ML/ folder | [ ] |
| Protocols | Classification service uses protocol | [ ] |
| Sendable | All ML services are Sendable | [ ] |
| Use Case | Identification logic in use case, not ViewModel | [ ] |
| DI Container | Classification service injected via container | [ ] |
| Error Types | Custom errors defined for ML failures | [ ] |
| Unit Tests | Use case has at least one unit test | [ ] |

### Performance Verification

| Metric | Target | Actual | Status |
|--------|--------|--------|--------|
| Model Load Time | < 2 seconds | | [ ] |
| Inference Time (avg) | < 500ms | | [ ] |
| Inference Time (max) | < 1000ms | | [ ] |
| Memory During Inference | < 200MB | | [ ] |
| Memory After Inference | Returns to baseline | | [ ] |
| App Size Increase | < 100MB (model) | | [ ] |

### Accuracy Verification

| Test Image | Expected Top Match | Actual Top Match | Confidence | Status |
|------------|-------------------|------------------|------------|--------|
| Rose photo | Rosa sp. | | | [ ] |
| Oak leaf | Quercus sp. | | | [ ] |
| Sunflower | Helianthus annuus | | | [ ] |
| Tulip | Tulipa sp. | | | [ ] |
| Fern | Pteridophyta sp. | | | [ ] |

---

## Phase 2 Completion Checklist

- [ ] All 10 tasks completed
- [ ] All functional tests pass
- [ ] All code quality checks pass
- [ ] All performance targets met
- [ ] Accuracy spot-checked with 5+ real plant images
- [ ] Core ML model included in app bundle
- [ ] PlantLabels.json contains all 1,081 species
- [ ] Performance results documented
- [ ] Code committed with descriptive message
- [ ] Ready for Phase 3 (PlantNet API Integration)

---

## Error Handling

### ML-Specific Errors
```swift
enum PlantClassificationError: Error, LocalizedError {
    case modelLoadFailed
    case imagePreprocessingFailed
    case inferenceTimeout
    case noResultsReturned
    case labelsNotFound

    var errorDescription: String? {
        switch self {
        case .modelLoadFailed:
            return "Unable to load plant identification model"
        case .imagePreprocessingFailed:
            return "Unable to process image for analysis"
        case .inferenceTimeout:
            return "Identification took too long"
        case .noResultsReturned:
            return "No plant species identified"
        case .labelsNotFound:
            return "Plant database not available"
        }
    }
}
```

---

## Notes

- Vision framework handles image scaling/cropping automatically via `imageCropAndScaleOption`
- Core ML models should be loaded once and reused (expensive to initialize)
- Use `MLModelConfiguration` to control compute units (CPU, GPU, Neural Engine)
- FLOAT16 precision reduces model size with minimal accuracy loss
- Test on real device early - simulator performance not representative
- Consider background thread for inference to keep UI responsive
- PlantNet-300K trained on European flora - accuracy varies by region

---

## Dependencies

| Dependency | Type | Notes |
|------------|------|-------|
| PlantNet300K.mlpackage | Core ML Model | ~50-100MB, bundled |
| PlantLabels.json | Data File | 1,081 species, bundled |
| Vision.framework | System | iOS 11+ |
| CoreML.framework | System | iOS 11+ |

---

## Risk Mitigation

| Risk | Mitigation |
|------|------------|
| Model too large for App Store | Use on-demand resources or model quantization |
| Inference too slow | Profile with Instruments, use Neural Engine |
| Low accuracy | Phase 3 adds API fallback for confirmation |
| Memory pressure | Unload model when not in use (trade-off: reload time) |
| Unsupported species | Show "Unknown" with low confidence, suggest API |