Turn workflows into requirements.
I built a 100-task evaluation set covering instruction following, consistency and revisions, then a 40-task short-drama set to examine narrative workflows more closely.
ByteDance
From creative intent
to product decisions.
How do we know an AI video agent is ready for real creative work?
A striking video is only part of the answer. A useful agent also needs to understand a brief, preserve creative intent across revisions, and deliver a coherent result.
At ByteDance, I worked on product strategy and evaluation for Seedance-powered video creation agents across Doubao and the Ark API. I translated creator workflows and product gaps into testable requirements, shared quality standards, and recommendations for product and algorithm teams.
- Role
- AI Product Strategist
- Skills
- Product research · Quality standards · Failure analysis
- Time
- 2026
- Tools
- Benchmarks · Scoring rubrics · Execution traces
My contribution I built and refined test cases, co-developed evaluation criteria and diagnostic labels, trained evaluators, and wrote comparison reports. Product and algorithm teams owned implementation and release decisions.
Diagrams and examples are recreated for this portfolio. Internal outputs, scores and performance data are omitted.
- Core evaluation set
- 100tasks
- Evaluation cycles supported
- 50+
- Evaluators using the guidelines
- 50+
The work connected three decisions.
Where is the gap?
Connect creative needs to product limitations.
What matters most?
Judge impact, recurrence and cause.
What should change?
Bring evidence and a direction to the team.
Start with what creators need to accomplish.
I studied competing agents and public creator workflows, including advertising films, then compared those needs with our agent’s capabilities.
Separate the symptom from the cause.
I reviewed finished videos first, then the agent’s recorded steps. This distinguished a poor creative result from the planning, tool or generation issue behind it.
Finding from my reviews
Creators needed control the tools did not offer.
I identified missing audio and visual-style anchors in the generation tools. This raised a concrete product question: what controls would help creators preserve their intent across a sequence?
Identify which problems deserve attention first.
I combined output usability, recurring failure patterns and scenario-level comparisons to support priority discussions. Diagnostic labels and execution traces helped connect each problem to a specific area for improvement.
Usability
Identify failures that prevent a usable creative output.
Recurrence
Group frequent issues and compare them within specific scenarios.
Actionability
Use traces to locate where a change could address the issue.
Example: why the failure stage changes the next step
If a presenter changes after an edit, a lost constraint points to planning; a correctly passed constraint with a mismatched output points to generation. I used this kind of trace analysis to narrow the investigation before recommending a direction.
Illustrative example reconstructed for this portfolio.
Turn a finding into a focused next step.
I brought representative cases and trace findings to product and algorithm teams, connecting the creator-facing problem to a specific area for improvement.
Subtitle failures → generation feedback
I flagged subtitle problems in generated videos to product and engineering. Locating the issue at the generation layer gave the team a specific capability to investigate.
Version differences → iteration decisions
I analyzed prompt and orchestration changes through comparisons of agent versions, models and competitors. Reports highlighted recurring weaknesses and scenario-level differences to inform optimization and model selection.
Product problem
What breaks in the creative workflow.
Supporting evidence
Recurring patterns, cases and trace findings.
Improvement direction
Where to intervene and what to compare next.
I carried this work into pre-release reviews for Doubao, preparing agent-versus-model comparisons to support the team’s release discussions.