A future-state global media data and AI architecture, designed in the open.
This is a solutioning lab for understanding, comparing, and refining options for a rapidly growing on-premises video estate — storage economics, global data movement, lifecycle management, and AI-driven discoverability of the archive.
Executive Overview
What this workspace is, the challenge it addresses, and the target state proposed for joint discussion.
Five forces are converging at once: storage economics at 44 PB and climbing, data gravity that makes cloud egress prohibitive, international movement between US and UK teams, time-to-content as the binding production constraint, and archive intelligence that turns retained footage back into usable creative material.
Store every asset on the appropriate economic tier. Move only the data that needs to move. Keep intelligence about every asset immediately available.
An on-premises-first, IBM-led hybrid architecture that preserves the existing NetApp investment while adding three things it does not have today: lifecycle economics across active, object, and archive tiers; global data movement between US, UK, and future international sites; and an AI intelligence layer that keeps every asset findable regardless of where the master physically sits.
Emphasis on options, decision points, the Hammerspace vs IBM Storage Scale bake-off, roadmap sequencing, and the discovery gaps both teams need to close together.
Business Problems
Nine pressures shaping the architecture, stated in the terms each audience cares about.
Rapid storage growth
Capacity is growing faster than a lean team can procure, rack, and operate it.
Growth compounds cost and management effort at the same time. Without tiering, every new petabyte is priced like production storage.
Data gravity and cloud egress economics
At this scale, moving data to and out of public cloud can cost more than storing it.
Cloud is not ruled out, but at 44 PB and rising, egress and retrieval charges dominate the model.
International transfer and collaboration
US and UK teams need working access to the same material without waiting on full copies.
Global production speed depends on how quickly a UK editor can start work, not on how fast a full transfer completes.
Shipping-drive risk and delay
Physical media moves add days, logistics load, and integrity exposure.
Drives in transit are unbilled time and unmanaged risk — loss, damage, or silent corruption.
Time-to-content
Time is the binding constraint across capture, ingest, edit, and delivery.
Every hour saved between capture and first edit is directly recoverable production capacity.
Archive discoverability
Historic footage is retained but hard to find at the moment of creative need.
Archive value only materializes if a producer can find the right shot in seconds instead of days.
8K growth
Higher resolution multiplies bytes per shooting day across every downstream tier.
Format change amplifies every other problem on this page simultaneously.
Lean operations team
~12 people support a growing global estate.
Any architecture that adds operational headcount is the wrong architecture.
Future multi-site expansion
Additional production and post locations are expected.
The design should make the second and third site an incremental step, not a re-architecture.
Current State
How content moves today, with confirmed facts separated from working assumptions.
Camera cards, up to ~300 TB/day at peak
Offload, checksum, and hand-off to production storage
Active editorial performance storage (NetApp)
Near-line / recently completed work
Retained content and longer-term holdings
Network transfer plus physical drive shipment for some UK movement
- ConfirmedUK post-production house involved in workflows
- ConfirmedDocumentary production alongside episodic content
- ConfirmedExisting NetApp footprint and support relationship
- ConfirmedPeak ingest up to ~300 TB/day
- ConfirmedTime described as the biggest operational enemy
- ConfirmedCloud viewed as potentially too expensive at this scale
- ConfirmedPhysical drive shipping used for some international movement
- AssumptionTier boundaries are capacity-based; policy thresholds not yet documented
- AssumptionUnique content vs protection copies inside 44 PB not yet separated
- AssumptionMAM/DAM and NLE stack details still to be gathered
- AssumptionUS↔UK circuit bandwidth and latency not yet measured
- AssumptionProxy generation point in the workflow to be confirmed
Target Architecture
A conceptual future-state model. Select any component to review its role, value, tradeoffs, and alternatives.
Technology Components
Every candidate component with role, benefits, pros, cons, dependencies, alternatives, and recommendation status.
Storage Lifecycle
Masters move down the tiers; intelligence about every asset stays online.
Camera cards and on-set capture land at high speed; checksums and proxies generated as early as workflow allows.
Full editorial performance while the project is being cut, reviewed, and delivered.
Delivery milestone triggers lifecycle evaluation rather than a manual cleanup task.
Master moves to object economics but remains immediately readable for re-cuts and repurposing.
Master preserved at the lowest sustainable cost; recall is a deliberate, tracked action.
The master moves down the tiers. The intelligence about the master — proxy, transcript, thumbnails, embeddings, technical metadata — stays online throughout, so discovery never depends on where the bytes live.
Editable assumptions
“Find the moment the doors open in the warehouse reveal”
Illustrative only — not a live search system and not connected to any archive. It shows how one result set can span tiers while every result remains discoverable.
"the moment the doors open" — transcript + scene match
visual embedding match, no dialogue
OCR on on-screen graphic + transcript
speaker + scene detection
Scenario Modeling
A client-side planning calculator. Change any assumption to see directional implications immediately.
Editable assumptions
The cost index is a unitless relative scale (active = 100 by default) used only to show directional differences between tiers. It is not pricing.
Projected capacity by tier
IllustrativeDirectional tier economics
DirectionalYear-5 estate priced entirely on the active tier vs the modelled tier distribution, using the relative cost index above.
Relative index only — not a cost saving estimate.
US↔UK transfer scenarios
Raw bandwidth mathTime to move a given volume at 10 Gbps with 70% usable efficiency. Excludes contention, protocol behaviour, and endpoint storage throughput.
Planning implication: a global namespace plus proxy-first workflows usually beats moving full masters — the transfer that matters is the one you avoid.
Modelled across 2 data centre location(s) with a 10-year retention assumption. Capacity shown is logical unique content; replication and protection copies are excluded until the storage economics assessment separates them.
Architecture Options
Three viable models compared against neutral decision criteria.
Option A — IBM-centric
IBM Storage Scale as the global data layer, IBM COS and Deep Archive for lifecycle, IBM data and AI plane throughout.
- One support and escalation path across storage, movement, and AI
- Tightest integration between tiers and the intelligence plane
- Simplest commercial and lifecycle alignment
- Less media-industry-specific tooling at the namespace layer
- Reduced flexibility to adopt best-in-class point solutions later
Option B — Best-of-breed
Hammerspace for the global data layer, NetApp retained for active production, mixed AI services selected per capability.
- Strong media-and-entertainment fit at the namespace and workflow layer
- Maximum freedom to swap components as the market moves
- Preserves the existing NetApp investment fully
- More vendors to integrate, support, and upgrade with ~12 people
- Integration and accountability sit with the customer unless contracted otherwise
Option C — Recommended hybrid
Working recommendationKeep NetApp for active production, run the Hammerspace vs IBM Storage Scale bake-off for the global layer, standardise on Aspera, IBM COS, Deep Archive + Diamondback, and the watsonx intelligence plane.
- No disruption to production storage on day one
- Economics improve immediately via object and archive tiers
- Global-layer choice is decided by evidence, not by architecture default
- Requires the bake-off to be run and concluded
- Two-vendor storage estate persists at least through the transition
Decision Support
The decisions that need to be made together, with recommendation, rationale, and the evidence required.
Global data layer: Hammerspace vs IBM Storage Scale
Open decision- US↔UK latency and bandwidth measurements
- Representative project transfer and open-in-NLE tests
- Operability walk-through with the infrastructure team
Future Tier-1 expansion approach
Open decision- 8K per-project footprint
- Tier-1 utilization and headroom
- AI pipeline concurrency targets
Hot / warm / cold policy thresholds
Open decision- Access frequency at 7 / 30 / 90 / 180 / 365 days
- Re-cut and repurposing frequency
Proxy strategy for AI processing
Open decision- Current proxy format and generation point
- Proxy quality sufficiency for OCR/VLM
Video AI services and model selection
Open decision- Labelled evaluation clips
- Accuracy and cost per hour of footage
- Language/accent coverage
Multi-site topology
Open decision- Confirmed future site list
- Per-site production profile
- Local vs central archive preference
WAN sizing and Aspera validation
Open decision- Circuit inventory
- Aspera test transfers on real project sizes
- Peak concurrency profile
POC / Roadmap
Three proposed validation workstreams that would replace assumptions with measured evidence.
Storage Economics Assessment
- Inventory NetApp products, raw vs usable capacity, and protection copies
- Separate unique content from replication and backup copies
- Collect access-pattern data at 7 / 30 / 90 / 180 / 365 days
- Model tier distribution against candidate lifecycle policies
- Agreed unique-content baseline
- Access-pattern curve accepted by both teams
- Directional tier-distribution model both teams believe
- Storage inventory exports
- Utilization and access telemetry
- Retention requirements
Decision input for lifecycle thresholds and the Tier-1 expansion approach.
Global Data Movement & Access Validation
- Measure existing circuit bandwidth, latency, and loss
- Run Aspera transfers using representative project sizes
- Bake-off Hammerspace and IBM Storage Scale on the same test workflow
- Time an editor opening a remote project end to end
- Measured time-to-first-edit for a remote project
- Transfer integrity verified across all test runs
- Bake-off scorecard completed against agreed criteria
- Network access and test windows
- Representative project data set
- Editor participation
Global data layer decision and WAN sizing recommendation.
Content Intelligence Pilot
- Select a representative corpus (for example, one season plus documentary B-roll)
- Run ASR, OCR, scene detection, and visual embedding over proxies
- Normalise and index through unstructured.io into watsonx.data
- Test editorial search use cases in watsonx.ai with real producers
- Agreed accuracy threshold on the evaluation set
- Producers find target shots faster than the current method
- Cost per hour of processed footage understood
- Pilot corpus and proxies
- Existing transcripts/metadata if available
- Producer/editor time
Model selection decision and a scoped plan for archive-wide indexing.
Open Questions
The discovery checklist both teams can work through together.