Add PlantGuide iOS app with plant identification and care management
- Implement camera capture and plant identification workflow - Add Core Data persistence for plants, care schedules, and cached API data - Create collection view with grid/list layouts and filtering - Build plant detail views with care information display - Integrate Trefle botanical API for plant care data - Add local image storage for captured plant photos - Implement dependency injection container for testability - Include accessibility support throughout the app Bug fixes in this commit: - Fix Trefle API decoding by removing duplicate CodingKeys - Fix LocalCachedImage to load from correct PlantImages directory - Set dateAdded when saving plants for proper collection sorting Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Trey t
327
Docs/Phase2_plan.md
Normal file
327
Docs/Phase2_plan.md
Normal file
@@ -0,0 +1,327 @@
|
||||
# Phase 2: On-Device ML
|
||||
|
||||
**Goal:** Offline plant identification with Core ML
|
||||
|
||||
**Prerequisites:** Phase 1 complete (camera capture working, folder structure in place)
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### 2.1 Download PlantNet-300K Pre-trained Weights
|
||||
- [ ] Create `scripts/` directory at project root
|
||||
- [ ] Download ResNet50 weights from Zenodo:
|
||||
```bash
|
||||
wget https://zenodo.org/records/4726653/files/resnet50_weights_best_acc.tar
|
||||
```
|
||||
- [ ] Verify file integrity (expected ~100MB)
|
||||
- [ ] Document download source and version in `scripts/README.md`
|
||||
|
||||
**Acceptance Criteria:** `resnet50_weights_best_acc.tar` file present and verified
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Convert Model to Core ML
|
||||
- [ ] Install Python dependencies:
|
||||
```bash
|
||||
pip install torch torchvision coremltools pillow numpy
|
||||
```
|
||||
- [ ] Create `scripts/convert_plantnet_to_coreml.py`:
|
||||
- Load ResNet50 architecture with 1,081 output classes
|
||||
- Load PlantNet-300K weights
|
||||
- Trace model with dummy input
|
||||
- Configure image input preprocessing (RGB, 224x224, normalized)
|
||||
- Convert to ML Program format
|
||||
- Target iOS 17+ deployment
|
||||
- Use FLOAT16 precision for performance
|
||||
- [ ] Run conversion script
|
||||
- [ ] Verify output `PlantNet300K.mlpackage` created successfully
|
||||
|
||||
**Acceptance Criteria:** `PlantNet300K.mlpackage` generated without errors, file size ~50-100MB
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Add Core ML Model to Xcode
|
||||
- [ ] Copy `PlantNet300K.mlpackage` to `ML/Models/`
|
||||
- [ ] Add to Xcode project target
|
||||
- [ ] Verify Xcode generates `PlantNet300K.swift` interface
|
||||
- [ ] Configure model to compile at build time (not runtime)
|
||||
- [ ] Test that project still builds successfully
|
||||
|
||||
**Acceptance Criteria:** Model visible in Xcode, auto-generated Swift interface available
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Create Plant Labels JSON
|
||||
- [ ] Download species list from PlantNet-300K dataset
|
||||
- [ ] Create `Resources/PlantLabels.json` with structure:
|
||||
```json
|
||||
{
|
||||
"labels": [
|
||||
{
|
||||
"index": 0,
|
||||
"scientificName": "Acer campestre",
|
||||
"commonNames": ["Field Maple", "Hedge Maple"],
|
||||
"family": "Sapindaceae"
|
||||
}
|
||||
],
|
||||
"version": "1.0",
|
||||
"source": "PlantNet-300K"
|
||||
}
|
||||
```
|
||||
- [ ] Ensure all 1,081 species mapped correctly
|
||||
- [ ] Validate JSON syntax
|
||||
- [ ] Create `ML/Services/PlantLabelService.swift` to load and query labels
|
||||
|
||||
**Acceptance Criteria:** JSON contains 1,081 species entries, loads without parsing errors
|
||||
|
||||
---
|
||||
|
||||
### 2.5 Implement Plant Classification Service
|
||||
- [ ] Create `ML/Services/PlantClassificationService.swift`
|
||||
- [ ] Define `PlantClassificationServiceProtocol`:
|
||||
```swift
|
||||
protocol PlantClassificationServiceProtocol: Sendable {
|
||||
func classify(image: CGImage) async throws -> [PlantPrediction]
|
||||
}
|
||||
```
|
||||
- [ ] Create `PlantPrediction` struct:
|
||||
- `speciesIndex: Int`
|
||||
- `confidence: Float`
|
||||
- `scientificName: String`
|
||||
- `commonNames: [String]`
|
||||
- [ ] Implement using Vision framework:
|
||||
- Create `VNCoreMLRequest` with PlantNet300K model
|
||||
- Configure `imageCropAndScaleOption` to `.centerCrop`
|
||||
- Process results from `VNClassificationObservation`
|
||||
- [ ] Return top 10 predictions sorted by confidence
|
||||
- [ ] Handle model loading errors gracefully
|
||||
|
||||
**Acceptance Criteria:** Service returns predictions for any valid CGImage input
|
||||
|
||||
---
|
||||
|
||||
### 2.6 Create Image Preprocessor
|
||||
- [ ] Create `ML/Preprocessing/ImagePreprocessor.swift`
|
||||
- [ ] Define `ImagePreprocessorProtocol`:
|
||||
```swift
|
||||
protocol ImagePreprocessorProtocol: Sendable {
|
||||
func prepare(image: UIImage) -> CGImage?
|
||||
func prepare(data: Data) -> CGImage?
|
||||
}
|
||||
```
|
||||
- [ ] Implement preprocessing pipeline:
|
||||
- Convert UIImage to CGImage
|
||||
- Handle orientation correction (EXIF)
|
||||
- Resize to 224x224 if needed (Vision handles this, but validate)
|
||||
- Convert color space to sRGB if needed
|
||||
- [ ] Add validation for minimum image dimensions
|
||||
- [ ] Handle edge cases (nil image, corrupt data)
|
||||
|
||||
**Acceptance Criteria:** Preprocessor handles images from camera and photo library correctly
|
||||
|
||||
---
|
||||
|
||||
### 2.7 Build Identify Plant On-Device Use Case
|
||||
- [ ] Create `Domain/UseCases/Identification/IdentifyPlantOnDeviceUseCase.swift`
|
||||
- [ ] Define use case protocol:
|
||||
```swift
|
||||
protocol IdentifyPlantOnDeviceUseCaseProtocol: Sendable {
|
||||
func execute(image: UIImage) async throws -> PlantIdentification
|
||||
}
|
||||
```
|
||||
- [ ] Implement use case:
|
||||
- Preprocess image
|
||||
- Call classification service
|
||||
- Map predictions to `PlantIdentification` entity
|
||||
- Set `source` to `.onDevice`
|
||||
- Record timestamp
|
||||
- [ ] Add to DIContainer factory methods
|
||||
- [ ] Create unit test with mock classification service
|
||||
|
||||
**Acceptance Criteria:** Use case integrates preprocessor and classifier, returns domain entity
|
||||
|
||||
---
|
||||
|
||||
### 2.8 Create Identification View
|
||||
- [ ] Create `Presentation/Scenes/Identification/IdentificationView.swift`
|
||||
- [ ] Create `Presentation/Scenes/Identification/IdentificationViewModel.swift`
|
||||
- [ ] Implement UI states:
|
||||
- Loading (during inference)
|
||||
- Success (show results)
|
||||
- Error (show retry option)
|
||||
- [ ] Display captured image at top
|
||||
- [ ] Show list of top 10 species matches
|
||||
- [ ] Add "Identify Again" button
|
||||
- [ ] Add "Save to Collection" button (disabled for Phase 2)
|
||||
- [ ] Navigate from CameraView after capture
|
||||
|
||||
**Acceptance Criteria:** View displays results after photo capture, handles all states
|
||||
|
||||
---
|
||||
|
||||
### 2.9 Build Species Match Components
|
||||
- [ ] Create `Presentation/Common/Components/SpeciesMatchCard.swift`:
|
||||
- Scientific name (primary text)
|
||||
- Common names (secondary text)
|
||||
- Confidence score
|
||||
- Ranking number (1-10)
|
||||
- Chevron for detail navigation (future)
|
||||
- [ ] Create `Presentation/Common/Components/ConfidenceIndicator.swift`:
|
||||
- Visual progress bar
|
||||
- Percentage text
|
||||
- Color coding:
|
||||
- Green: > 70%
|
||||
- Yellow: 40-70%
|
||||
- Red: < 40%
|
||||
- [ ] Style components with consistent design language
|
||||
- [ ] Ensure accessibility labels set correctly
|
||||
|
||||
**Acceptance Criteria:** Components render correctly with sample data, accessible
|
||||
|
||||
---
|
||||
|
||||
### 2.10 Performance Testing
|
||||
- [ ] Create `ML/Tests/ClassificationPerformanceTests.swift`
|
||||
- [ ] Measure inference time on real device:
|
||||
- Test with 10 different plant images
|
||||
- Record min/max/average times
|
||||
- Target: < 500ms average
|
||||
- [ ] Measure memory usage during inference:
|
||||
- Target: < 200MB peak
|
||||
- [ ] Test on oldest supported device (if available)
|
||||
- [ ] Profile with Instruments (Core ML template)
|
||||
- [ ] Document results in `Docs/Phase2_performance_results.md`
|
||||
|
||||
**Acceptance Criteria:** Average inference < 500ms on target device
|
||||
|
||||
---
|
||||
|
||||
## End-of-Phase Validation
|
||||
|
||||
### Functional Verification
|
||||
|
||||
| Test | Steps | Expected Result | Status |
|
||||
|------|-------|-----------------|--------|
|
||||
| Model Loads | Launch app | No model loading errors in console | [ ] |
|
||||
| Labels Load | Launch app | PlantLabels.json parsed successfully | [ ] |
|
||||
| Camera → Identify | Take photo, wait | Identification results appear | [ ] |
|
||||
| Results Display | View results | 10 species shown with confidence scores | [ ] |
|
||||
| Confidence Colors | View varied results | Colors match confidence levels | [ ] |
|
||||
| Loading State | Take photo | Loading indicator shown during inference | [ ] |
|
||||
| Error Handling | Force error (mock) | Error view displays with retry | [ ] |
|
||||
| Retry Flow | Tap retry | Returns to camera | [ ] |
|
||||
|
||||
### Code Quality Verification
|
||||
|
||||
| Check | Criteria | Status |
|
||||
|-------|----------|--------|
|
||||
| Build | Project builds with zero warnings | [ ] |
|
||||
| Architecture | ML code isolated in ML/ folder | [ ] |
|
||||
| Protocols | Classification service uses protocol | [ ] |
|
||||
| Sendable | All ML services are Sendable | [ ] |
|
||||
| Use Case | Identification logic in use case, not ViewModel | [ ] |
|
||||
| DI Container | Classification service injected via container | [ ] |
|
||||
| Error Types | Custom errors defined for ML failures | [ ] |
|
||||
| Unit Tests | Use case has at least one unit test | [ ] |
|
||||
|
||||
### Performance Verification
|
||||
|
||||
| Metric | Target | Actual | Status |
|
||||
|--------|--------|--------|--------|
|
||||
| Model Load Time | < 2 seconds | | [ ] |
|
||||
| Inference Time (avg) | < 500ms | | [ ] |
|
||||
| Inference Time (max) | < 1000ms | | [ ] |
|
||||
| Memory During Inference | < 200MB | | [ ] |
|
||||
| Memory After Inference | Returns to baseline | | [ ] |
|
||||
| App Size Increase | < 100MB (model) | | [ ] |
|
||||
|
||||
### Accuracy Verification
|
||||
|
||||
| Test Image | Expected Top Match | Actual Top Match | Confidence | Status |
|
||||
|------------|-------------------|------------------|------------|--------|
|
||||
| Rose photo | Rosa sp. | | | [ ] |
|
||||
| Oak leaf | Quercus sp. | | | [ ] |
|
||||
| Sunflower | Helianthus annuus | | | [ ] |
|
||||
| Tulip | Tulipa sp. | | | [ ] |
|
||||
| Fern | Pteridophyta sp. | | | [ ] |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Completion Checklist
|
||||
|
||||
- [ ] All 10 tasks completed
|
||||
- [ ] All functional tests pass
|
||||
- [ ] All code quality checks pass
|
||||
- [ ] All performance targets met
|
||||
- [ ] Accuracy spot-checked with 5+ real plant images
|
||||
- [ ] Core ML model included in app bundle
|
||||
- [ ] PlantLabels.json contains all 1,081 species
|
||||
- [ ] Performance results documented
|
||||
- [ ] Code committed with descriptive message
|
||||
- [ ] Ready for Phase 3 (PlantNet API Integration)
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### ML-Specific Errors
|
||||
```swift
|
||||
enum PlantClassificationError: Error, LocalizedError {
|
||||
case modelLoadFailed
|
||||
case imagePreprocessingFailed
|
||||
case inferenceTimeout
|
||||
case noResultsReturned
|
||||
case labelsNotFound
|
||||
|
||||
var errorDescription: String? {
|
||||
switch self {
|
||||
case .modelLoadFailed:
|
||||
return "Unable to load plant identification model"
|
||||
case .imagePreprocessingFailed:
|
||||
return "Unable to process image for analysis"
|
||||
case .inferenceTimeout:
|
||||
return "Identification took too long"
|
||||
case .noResultsReturned:
|
||||
return "No plant species identified"
|
||||
case .labelsNotFound:
|
||||
return "Plant database not available"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Vision framework handles image scaling/cropping automatically via `imageCropAndScaleOption`
|
||||
- Core ML models should be loaded once and reused (expensive to initialize)
|
||||
- Use `MLModelConfiguration` to control compute units (CPU, GPU, Neural Engine)
|
||||
- FLOAT16 precision reduces model size with minimal accuracy loss
|
||||
- Test on real device early - simulator performance not representative
|
||||
- Consider background thread for inference to keep UI responsive
|
||||
- PlantNet-300K trained on European flora - accuracy varies by region
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Type | Notes |
|
||||
|------------|------|-------|
|
||||
| PlantNet300K.mlpackage | Core ML Model | ~50-100MB, bundled |
|
||||
| PlantLabels.json | Data File | 1,081 species, bundled |
|
||||
| Vision.framework | System | iOS 11+ |
|
||||
| CoreML.framework | System | iOS 11+ |
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Model too large for App Store | Use on-demand resources or model quantization |
|
||||
| Inference too slow | Profile with Instruments, use Neural Engine |
|
||||
| Low accuracy | Phase 3 adds API fallback for confirmation |
|
||||
| Memory pressure | Unload model when not in use (trade-off: reload time) |
|
||||
| Unsupported species | Show "Unknown" with low confidence, suggest API |
|
||||
Reference in New Issue
Block a user