What to know
- A 55% rubric average is not a 55% full-task completion rate.
- Our interpretation: ask which missing requirements would prevent you from accepting the output.
- For a learning project, document the acceptance decision alongside the finished workflow.
What OpenAI reported
OpenAI’s October 6 account compares GPT-6 Astra at Max reasoning with GPT-5.6 Sol at High on 11 selected Ironclad tasks. Mean rubric scores were 55.0% and 41.6%, respectively. Tasks concerned agreements, approvals, and reusable clauses; each had 8–50 scoring criteria. Astra was trained on Ironclad tasks.
Reported timing is simulated, not measured customer savings. Synthetic training tasks used filtered public SEC contracts. The comparison does not isolate model changes from reasoning settings.
Sources: OpenAI
Our interpretation: inspect the unfinished requirements
A useful question for a legal operations team is what would make an otherwise polished workflow unacceptable. We would distinguish a cosmetic omission from a missing control before deciding how much assistance to delegate. An aggregate score cannot make that acceptance decision for a particular organization.
Consider a hypothetical training exercise using an invented agreement. A colleague asks you to update one clause while preserving the approved version for existing agreements. Build a small demonstration, then deliberately test an older agreement and a newly created one. Record which text each receives, who approved the change, and how you would reverse it. This is our suggested exercise, not a task or result documented in the evaluation.
For a paralegal or operations specialist, the portfolio artifact could be an acceptance checklist and a short explanation of an error you caught. For a developer, it could include reproducible checks and a change log. The point is to make your reasoning inspectable: another person should understand why you accepted a result and where your authority ended. That is a concrete learning objective; we are not predicting a hiring advantage or a change in demand for either profession.
Sources & method
AI-generated synthesis of OpenAI’s own research evaluation, not independently replicated. Selected workflows cannot establish general legal performance, staffing effects, or reliability on every required step. Practical suggestions are our interpretation.
- Advancing computer use with Ironclad OpenAI ·
Drafted and source-checked by Codex AI agents. Read our sourcing, illustration, and correction policy.
Explore the occupations
What could this mean for your work?
See task exposure and potential career moves for roles connected to this story.
