233 lines
16 KiB
Markdown
233 lines
16 KiB
Markdown
# Plan: VNC Client App (Screens-style) — iPhone + iPad
|
||
|
||
## Context
|
||
|
||
The user wants to build a VNC remote-desktop app for iPhone and iPad modeled on [Screens by Edovia](https://www.edovia.com/en/screens/), which is the market leader in the category. Screens exposes a premium, polished VNC experience: Bonjour discovery, saved connections, local + SSH tunneled transport, Touch Mode / Trackpad Mode input, clipboard sync, multi-monitor, hardware keyboard + Apple Pencil, and (in the full product) a relay service and file transfer.
|
||
|
||
Working directory `/Users/treyt/Desktop/code/Screens` is empty — this is a greenfield build. Scope agreed with the user:
|
||
|
||
- **Platforms**: iPhone + iPad (iOS/iPadOS). No Mac / visionOS in phase 1.
|
||
- **RFB engine**: [RoyalVNCKit](https://github.com/royalapplications/royalvnc) (MIT, Swift, supports Raw/CopyRect/RRE/CoRRE/Hextile/Zlib/ZRLE/Tight + VNC password + Apple Remote Desktop auth). iOS UIView rendering is flagged "work in progress" upstream, so we own the UIKit framebuffer view.
|
||
- **MVP ambition**: "Feature-parity push" — MVP + clipboard sync, multi-monitor, trackpad mode, Apple Pencil. ~2 months.
|
||
- **Remote access strategy**: Tailscale. A device on a tailnet is reached by its Tailscale IP or MagicDNS name exactly like a LAN host, and the transport is already WireGuard-encrypted end-to-end. The app does not need to bundle SSH tunneling or a relay — we just connect TCP and let Tailscale handle the network layer.
|
||
- **No account system / subscription** in phase 1. Saved connections sync via CloudKit (iCloud) if the user opts in.
|
||
|
||
Intended outcome: a shippable, App Store-quality VNC client that a Mac/Linux/RPi user can drive from an iPhone or iPad on their LAN or through SSH.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────┐
|
||
│ App target (thin shell, ~10% of code) │
|
||
│ VNCApp.swift · RootView · AppStateController │
|
||
└──────────────────────┬───────────────────────────────────────┘
|
||
│ environment
|
||
┌──────────────────────▼───────────────────────────────────────┐
|
||
│ VNCUI (Swift package) │
|
||
│ ConnectionListView · AddConnectionView · SessionView │
|
||
│ FramebufferView (UIViewRepresentable → FramebufferUIView) │
|
||
│ InputMapper (Touch/Trackpad/HW keyboard/Pencil) │
|
||
└──────────────────────┬───────────────────────────────────────┘
|
||
│
|
||
┌──────────────────────▼───────────────────────────────────────┐
|
||
│ VNCCore (Swift package, @testable without simulator) │
|
||
│ SessionController ◀─────── Transport protocol │
|
||
│ │ └── DirectTransport (NW) │
|
||
│ ▼ │
|
||
│ RoyalVNCKit · DiscoveryService (NWBrowser _rfb._tcp) │
|
||
│ ConnectionStore (SwiftData + Keychain refs) │
|
||
│ KeychainService · ClipboardBridge │
|
||
└──────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
`Transport` stays behind a protocol so a future relay or optional SSH-tunnel add-on can slot in without touching `SessionController`.
|
||
|
||
### Why this shape
|
||
|
||
- **Thin `@main`** and `AppStateController`: the app has distinct states (launching → list → connecting → connected → error). Modeling them as an enum avoids "boolean soup" and keeps logic testable. Pattern from the `axiom:axiom-app-composition` skill (Part 1).
|
||
- **VNCCore is a Swift Package** with no UI imports. This lets us run the protocol + transport tests with `swift test` (~sub-second) instead of `xcodebuild test`. 60× faster TDD loop.
|
||
- **Transport protocol** isolates the network layer from the session layer. Phase-1 has one implementation (`DirectTransport` over `NWConnection`), which already handles both LAN and Tailscale — a tailnet host resolves and routes like any other host from the app's perspective. If we ever add SSH tunneling or a rendezvous relay, they drop in as additional `Transport` conformers.
|
||
- **UIKit-backed framebuffer view** wrapped in `UIViewRepresentable`. SwiftUI's `Canvas` is too slow for 30+ fps blits at ~2M pixels. A `CALayer` with `contents = IOSurface`-backed `CGImage` (or a small Metal view if ZRLE/Tight decode runs on GPU later) gives us consistent 60 Hz.
|
||
|
||
## Key Dependencies
|
||
|
||
| Package | Purpose | License |
|
||
| --- | --- | --- |
|
||
| [royalvnc](https://github.com/royalapplications/royalvnc) | RFB protocol, encodings, auth | MIT |
|
||
| Apple first-party: `Network`, `SwiftData`, `CloudKit`, `Security` (Keychain), `UIKit`, `SwiftUI`, `Combine` | n/a | n/a |
|
||
|
||
Tailscale requires no SDK integration — it runs as a system-wide VPN/Network Extension installed separately by the user. Once active, remote hosts are reachable by Tailscale IP or MagicDNS name from our plain `NWConnection`.
|
||
|
||
Deployment target: **iOS 18.0**. Gets us `@Observable` macro, SwiftData stability fixes, and modern Network.framework APIs. Skip iOS 26-only features (Liquid Glass components) behind `if #available` gates so we don't block on Xcode 26 availability.
|
||
|
||
## Data Model
|
||
|
||
SwiftData models live in `VNCCore`:
|
||
|
||
```swift
|
||
@Model final class SavedConnection {
|
||
@Attribute(.unique) var id: UUID
|
||
var displayName: String
|
||
var host: String // hostname, IPv4, or IPv6
|
||
var port: Int // default 5900
|
||
var colorTag: ColorTag // UI identification
|
||
var lastConnectedAt: Date?
|
||
var preferredEncodings: [String] // ordered, persisted for reconnect
|
||
var keychainTag: String // opaque ref to VNC password entry
|
||
var quality: QualityPreset // .adaptive / .high / .low
|
||
var viewOnly: Bool
|
||
var curtainMode: Bool
|
||
}
|
||
```
|
||
|
||
VNC passwords go to Keychain (`kSecClassGenericPassword` with `kSecAttrAccessible = kSecAttrAccessibleWhenUnlockedThisDeviceOnly`). The `keychainTag` on the model is the account name — we never persist the secret in SwiftData. Optional CloudKit sync of `SavedConnection` (minus secrets) via SwiftData's CloudKit integration.
|
||
|
||
## Phased Delivery
|
||
|
||
### Phase 0 — Scaffold (~3 days)
|
||
- Create Xcode project (working name — App Store name TBD to avoid trademark conflict; brainstorm candidates before public launch).
|
||
- SPM packages `VNCCore` and `VNCUI` inside the repo.
|
||
- Add RoyalVNCKit as a dependency.
|
||
- Thin `@main`, `AppStateController` with `case launching / list / connecting(SavedConnection) / session(SessionController) / error(AppError)`, `RootView` switch.
|
||
- CI via `xcodebuild test` on `VNCUI` + `swift test` on `VNCCore`.
|
||
- App icon + launch screen placeholder.
|
||
|
||
### Phase 1 — MVP connect + view + tap (~2 weeks)
|
||
1. `DiscoveryService` using `NWBrowser` on `_rfb._tcp` and `_workstation._tcp` (macOS Screen Sharing advertises the latter).
|
||
2. `ConnectionStore` — SwiftData CRUD + Keychain round-trip.
|
||
3. `ConnectionListView`, `AddConnectionView`, basic Liquid Glass cards when available.
|
||
4. `DirectTransport` using `NWConnection` (TCP, TLS optional for VeNCrypt later).
|
||
5. `SessionController` wrapping `RoyalVNCKit.VNCConnection`: lifecycle, framebuffer updates, reconnection.
|
||
6. `FramebufferUIView` (UIKit): `CALayer` whose `contents` is a `CGImage` built from the RoyalVNC framebuffer bytes. Handle partial rect updates (dirty-rect blits, not full redraws).
|
||
7. Touch Mode input: tap → left click, two-finger tap → right click, pan → scroll. Nothing fancy yet.
|
||
8. VNC password auth + "save to Keychain" flow.
|
||
|
||
**Exit criterion**: from a cold launch I can discover my Mac via Bonjour, enter the Screen Sharing password, see the desktop, click a Finder icon, and type into Spotlight.
|
||
|
||
### Phase 2 — Input parity (~1.5 weeks)
|
||
- Trackpad Mode: a floating soft cursor SwiftUI overlay; pan moves the cursor, tap-to-click, pinch to zoom into the framebuffer. Momentum scroll.
|
||
- Hardware keyboard via `pressesBegan`/`pressesEnded` on the framebuffer view — full modifier + function key support (⌘, ⌥, ⌃, ⇧, F1–F20, arrows).
|
||
- Magic Trackpad 2 / Magic Mouse: `UIPointerInteraction` + indirect pointer events → VNC pointer events.
|
||
- Adaptive quality: choose encoding (Tight vs ZRLE) and JPEG quality based on `NWPathMonitor` link type + measured RTT.
|
||
- Reconnect backoff with jitter.
|
||
|
||
### Phase 3 — Productivity features (~2 weeks)
|
||
- Clipboard sync: bidirectional `UIPasteboard` ↔ VNC `ClientCutText`/`ServerCutText`. Handle Unicode (ISO-8859-1 limitation of classic spec — prefer extended clipboard pseudo-encoding where the server advertises it). Opt-in toggle per connection and a global kill switch for privacy.
|
||
- Multi-monitor: parse `DesktopSize` / `ExtendedDesktopSize` pseudo-encodings; offer a monitor picker that sends `SetDesktopSize` or pans the viewport. Remember last selection per connection.
|
||
- Apple Pencil: treat as pointer with pressure — if the remote is a Mac running a drawing app, pass pressure via a custom extension if available, otherwise plain pointer. Hover support on M2 iPad Pro.
|
||
- Screenshot capture: grab current framebuffer bytes → `CGImage` → share sheet.
|
||
- "On disconnect" actions: prompt / reconnect / return to list.
|
||
- View Only and Curtain Mode toggles wired into RoyalVNC's session options.
|
||
|
||
### Phase 4 — Polish + ship (~1.5 weeks)
|
||
- iPad multi-window: `WindowGroup(for: SavedConnection.ID.self)` so each connection can open in its own window, each with its own `SessionController` instance.
|
||
- CloudKit sync of `SavedConnection` records via SwiftData's `ModelConfiguration(cloudKitDatabase: .private("iCloud..."))`. Secrets stay local-Keychain only.
|
||
- Accessibility pass (VoiceOver, Dynamic Type on non-framebuffer chrome, Reduce Motion).
|
||
- Empty states, error recovery, network-change handling (Wi-Fi → cellular with session pause/resume).
|
||
- Privacy manifest, App Store screenshots, TestFlight.
|
||
|
||
## Critical Files to Create
|
||
|
||
```
|
||
Screens.xcodeproj
|
||
Screens/
|
||
App/
|
||
VNCApp.swift # @main (thin)
|
||
RootView.swift
|
||
AppStateController.swift
|
||
Resources/
|
||
Assets.xcassets
|
||
Info.plist
|
||
|
||
Packages/VNCCore/
|
||
Package.swift
|
||
Sources/VNCCore/
|
||
Session/
|
||
SessionController.swift # wraps RoyalVNCKit.VNCConnection
|
||
SessionState.swift # enum of connection states
|
||
Transport/
|
||
Transport.swift # protocol
|
||
DirectTransport.swift # NWConnection
|
||
SSHTunnelTransport.swift # SwiftNIOSSH
|
||
Discovery/
|
||
DiscoveryService.swift # NWBrowser on _rfb._tcp + _workstation._tcp
|
||
Storage/
|
||
SavedConnection.swift # @Model
|
||
ConnectionStore.swift # actor-wrapped ModelContext helper
|
||
Security/
|
||
KeychainService.swift # SecItem wrapper
|
||
Clipboard/
|
||
ClipboardBridge.swift
|
||
Tests/VNCCoreTests/
|
||
TransportTests.swift
|
||
SessionControllerTests.swift # mock Transport
|
||
ConnectionStoreTests.swift
|
||
|
||
Packages/VNCUI/
|
||
Package.swift
|
||
Sources/VNCUI/
|
||
List/
|
||
ConnectionListView.swift
|
||
ConnectionCard.swift
|
||
Edit/
|
||
AddConnectionView.swift
|
||
Session/
|
||
SessionView.swift # hosts the framebuffer + toolbars
|
||
FramebufferView.swift # UIViewRepresentable
|
||
FramebufferUIView.swift # UIKit CALayer-backed renderer
|
||
InputMapper.swift # touch/trackpad/keyboard → RFB events
|
||
TrackpadCursorOverlay.swift
|
||
ToolbarView.swift
|
||
Settings/
|
||
SettingsView.swift
|
||
```
|
||
|
||
## Reusable Pieces (don't rebuild)
|
||
|
||
- **RoyalVNCKit** gives us the entire RFB state machine, encodings, auth. We *consume* it; we do not fork.
|
||
- **`NWBrowser` + `NWListener`** from Network.framework — no need for third-party Bonjour.
|
||
- **`AppStateController` pattern** from `axiom:axiom-app-composition` — copy the boilerplate for loading/list/session/error states.
|
||
- **`axiom:axiom-swiftui-nav`** for the iPad split-view connection-list-to-session navigation.
|
||
- **`axiom:axiom-uikit-bridging`** for `UIViewRepresentable` coordinator patterns on `FramebufferUIView`.
|
||
- **`axiom:axiom-keychain`** for the `SecItem` wrapper.
|
||
- **`axiom:axiom-networking`** for `NWConnection` TLS/retry patterns reused inside `DirectTransport`.
|
||
|
||
## Verification
|
||
|
||
Layered testing, unit-heavy because the simulator is slow.
|
||
|
||
1. **VNCCore unit tests (`swift test`, ~sub-second)**
|
||
- `TransportTests`: feed a synthetic RFB handshake byte stream through a `MockTransport`, assert `SessionController` transitions.
|
||
- `ConnectionStoreTests`: in-memory SwiftData container, verify CRUD + Keychain round-trip (Keychain stubbed via protocol in tests).
|
||
- `DiscoveryServiceTests`: inject a fake `NWBrowser` surface, verify published result list.
|
||
|
||
2. **VNCUI UI snapshot / interaction tests (`xcodebuild test`)**
|
||
- `InputMapperTests`: given a `CGPoint` in framebuffer coordinates and a gesture, assert the emitted RFB pointer/key events.
|
||
- Preview-driven manual checks for `ConnectionListView`, `AddConnectionView` on iPhone + iPad size classes.
|
||
|
||
3. **End-to-end manual matrix** (run before TestFlight each phase):
|
||
- Mac (macOS Screen Sharing, port 5900) — discovery, password, Touch Mode click through, clipboard.
|
||
- Linux (`tigervnc-server` in Docker) — ZRLE encoding, resolution change during session.
|
||
- Raspberry Pi (`realvnc-vnc-server`) — Raspberry Pi Connect auth variant, low-bandwidth adaptive quality.
|
||
- Tailscale remote case: the same three above with the iPhone on cellular reaching the host by its MagicDNS name.
|
||
- iPad multi-window: two simultaneous sessions to different hosts.
|
||
- Cellular → Wi-Fi handoff mid-session (airplane-mode toggle).
|
||
|
||
4. **Performance budget** — on iPhone 15, a 2560×1440 @ 30 fps ZRLE stream should stay under 25% CPU and 40% battery-per-hour. Profile with Instruments Time Profiler + Energy Log each phase. See `axiom:axiom-performance-profiling`.
|
||
|
||
5. **App Review preflight**: privacy manifest for `NSUserDefaults`, `UIPasteboard`, and network access; screenshots per device class; `axiom:axiom-app-store-submission` checklist before first TestFlight.
|
||
|
||
## Open Questions to Resolve Before Phase 0
|
||
|
||
- **App Store name**: "Screens" is owned by Edovia — we cannot ship under that name. Pick a working title (e.g. "Pikelet", "Portal VNC", "Farcast") before public artifacts.
|
||
- **Bundle identifier + team ID**: confirm Apple Developer account + agreed bundle prefix.
|
||
- **App icon + brand direction**: can be deferred to Phase 5 but worth sketching early.
|
||
|
||
## Sources
|
||
|
||
- [Screens by Edovia — marketing site](https://www.edovia.com/en/screens/)
|
||
- [RoyalVNCKit (royalapplications/royalvnc)](https://github.com/royalapplications/royalvnc)
|
||
- [RFB protocol — Wikipedia](https://en.wikipedia.org/wiki/RFB_protocol) · [RFC 6143 (RFB 3.8)](https://datatracker.ietf.org/doc/html/rfc6143)
|
||
- [Tailscale on iOS](https://tailscale.com/kb/1020/install-ios)
|
||
- Axiom skills consulted: `axiom-app-composition`, `axiom-uikit-bridging`, `axiom-networking`, `axiom-keychain`, `axiom-swiftui-nav`, `axiom-performance-profiling`, `axiom-app-store-submission`.
|