On Sunday we sent you a recap of key moments from Thesis: 2027, our first conference. Today we’re taking a closer look at Dan Shipper’s closing keynote, which argues that when everyone has the same AI models, what matters is your judgment about what’s good. Later, head of evals Mike Taylor shows you how to write down that judgment so you can test AI against it, and staff writer Laura Entis shares what our speakers told her backstage.—Kate Lee
Was this newsletter forwarded to you? Sign up to get it in your inbox.
‘The Every Podcast’: Where humans belong in an AI future
What does great human work look like after automation?
Last week, we wrapped up Thesis: 2027, our conference dedicated to answering this question. In his closing keynote, Dan Shipper argues that figuring out what comes next for humans means predicting the future, and there are two ways to do that. The “aggregate” draws a trend line from troves of data. The “intimate” zooms in on one real situation and extrapolates from there. Seen through the aggregate, AI looks set to replace us, but up close, it looks very different. The intimate lens is how Every sees the future of work.
On this week’s episode of The Every Podcast, we’re sharing Dan’s full keynote, where he explains why benchmark charts make the future of work look bleaker than it is, why picking the right idea still takes a human, and how we should really evaluate models.
Here are the highlights:
- Why the big-picture data makes the AI future look more daunting than it is. Many predictions about AI and work draw from the aggregate. On OpenAI’s GDPval benchmark, models went from beating human experts 12.4 percent of the time in 2024 to about 83 percent in 2026. In one week, Dan did the work of five 2020-era hires with AI agents: analyzing survey data, creating a mini-video game, spotting a sales opportunity, vibe coding a CRM, and using it to do cold outreach. “I got all of this for free as part of my Claude subscription,” he says. But the aggregate perspective has limits: To measure model progress, it pits AI against humans working without it. “It has to freeze you—freeze humans in 2020,” Dan says. On the other hand, predictions drawing from the intimate zoom in on how people actually work. As the models improve, so do the people using them, so human work doesn’t trend toward zero. It rides on top of the same curve. This approach, Dan says, “empowers you to see what you might build and who you might become.”
- Why human perspective is the currency of the AI era. When members of Every’s team ran the same customer survey through Claude, each person found something different. Go-to-market engineer Yash Poojary saw new subscribers churning because they think Every sends too much email. Head of marketing Douglas Brundage saw that readers feel isolated because few people around them share their enthusiasm for AI. Claude could generate the insights Yash and Douglas found, but it would have trouble deciding what to prioritize. That takes what only a person brings: a “non-consensus perspective” and responsibility for a particular corner of the world. “Generation is really cheap, but ranking is hard,” Dan says.
- We’re grading AI on the wrong test. Once everyone has access to the same models, raw intelligence stops being an advantage. Yet today’s benchmarks still focus on math and coding problems with clear right answers that are easy to validate. That’s exactly why models have gotten so good at them and why those skills have become commodities. Dan argues that we should ask a different question: “How good is it at doing my work from my perspective?” Once you measure models that way, you start to see model progress “not as something that’s going to replace you, but something that works for you.”
This is a must-watch for anyone wondering where humans fit in a future shaped by AI. 🎧 🖥 Listen on Spotify or Apple Podcasts, watch on X or YouTube, or read the transcript.
Miss an episode? Catch up on recent conversations with Sam Altman; Emmy-winning TV writer Fred Graver; Hugging Face’s first investor, Matt Hartman; Granola founder Chris Pedregal; writer Craig Mod; GitHub COO Kyle Daigle; Anthropic head of product Mike Krieger; and others to learn how they use AI to think, create, and relate.—Miriam Partington
Skill share: Write your own test for AI
Dan’s keynote ends on a challenge: Stop asking how a model scores on the industry’s benchmarks and start asking how well it does your work. At Thesis, head of evals Mike Taylor spent his Classroom session showing how to build that test without being an evals expert. “You don’t have to be very good at evals,” he told the room. “You just have to have an opinion.”
What it does. Mike’s method turns the corrections you already make into a personal benchmark: a set of tests for how well AI does your work, by your standards. Say you ask an AI for a deck that teaches a team Compound Engineering, Every’s method for coding with AI agents, and it reuses an illustration from your brand guide. You say, “Always make a custom SVG” (scalable vector graphic, a format for crisp diagrams). That note becomes a check, a yes-or-no question an AI judge can ask of any future deck: Did it draw its own diagram? In turn, the checks let you compare models on your standards rather than the industry’s. When Mike ran this task across models, GPT 6.1 Sol failed, reusing the image from the brand guide. Grok 4.6, which he hadn’t expected much from, passed by drawing its own.
Why talking works. You don’t need to write a rubric. Saying what you like and don’t like is enough because “AI is really good at taking that unstructured transcript and turning it into evals,” Mike says. Say it out loud if you can: “We found voice mode works best for this.”
The catch: overfitting. Checks can also teach the wrong lesson. When Mike told a model its ASCII-art cat “isn’t recognizable as a cat,” it wrote a fair check, but it also gave itself a rule: “the cat sits facing front.” So when Mike asked for a cat curled up asleep, it drew a front-facing face anyway. The drawing passed the check and still wasn’t what he asked for. Mike calls this kind of overfitting “the number one problem in AI right now.” Read your checks the way you’d read a draft and keep testing on the work you do every day. As Mike puts it, “Reality is the ultimate eval.”
Steal it. Paste this into any chat AI the next time you’re correcting an output for a recurring task:
You’ll need: Any chat AI. Voice dictation helps.
Go deeper: Mike’s guide to building your own personal AI benchmark and his full session in the Thesis recordings (Classroom, 0:33:43).
Backstage at Thesis
Between sessions, staff writer Laura Entis pulled 17 of our speakers into a side room and asked them what they know about working with AI that most people don’t. Here are three of their answers (with more to come in later issues).
Counterpoint: Where the hours saved by AI agents actually go
The claim: AI will wipe out entry-level jobs and push us into a recession. Read the forecasts and the benchmarks, and that’s easy to believe.
The view from inside: Hugh Wimberly, a staff software engineer at Ode (the enterprise AI services firm Anthropic launched with Blackstone and Hellman & Friedman) who spoke at Thesis, builds AI agents inside client companies. Before taking the job, he asked whether Ode took contracts “just trying to eliminate jobs.” Six clients later, his projects haven’t triggered layoffs anywhere. Two examples:
- The onboarding team. At one business-to-business company, setting up each new customer took “up to about 100 hours,” most of which was “just clicking around in a UI.” An agent took over almost all the clicking, cutting onboarding time to about 30 hours. But automating 70 percent of the work didn’t lead to cutting 70 percent of the team. Cheaper onboarding let the team go after customers who couldn’t previously afford its services, so “even though this team is three times as productive, we still want more employees than we had before.”
- The accountants. Wimberly’s current client does accounting for very large businesses, work that now takes less time. Its partners already hold their clients’ financial statements and tax returns, and they golf and have dinner with them. So the firm is exploring having agents search those records for novel opportunities, like telling a client with a chain of businesses where “it might actually be really valuable for you to open up a new restaurant.” This transforms the accountant into something closer to “a full-fledged financial advisor.”
Wimberly doesn’t sugarcoat this work: “Some individual jobs are going away,” including, at least twice on his projects, offshore jobs that consisted of clicking the same button over and over. And his sample is self-selected: His clients’ teams help drive the change rather than having it imposed on them.
Try it this week: Give an agent what you already have on the client, boss, or team you know best, such as reports, meeting notes, and emails (but check what you’re allowed to share first). Ask:
Bring the best answer to your next meeting. The goal: Hand off the work anyone could do and grow the part only you can do, which is knowing the people you serve well enough to spot what they need next.
Prediction: In 2027, your job is teaching AI
The prediction: Katy George, Microsoft’s corporate vice president of AI and work transformation, sees AI at work as moving in three stages. It started as a “set of personal productivity tools.” Now it’s changing “the entire way the organization works at scale.” Next comes “a continuous learning organization,” an idea management consultants and HR people have talked about for years, now “codified into AI.” For people, that means “not doing stuff ourselves, but actually teaching AI to do it,” she says, “continually and constantly raising the bar.”
Timeline: 2027.
Our take: We buy it. It’s Dan’s keynote seen directly from inside a company of 230,000. If AI makes generating cheap, people’s jobs become deciding what good looks like and teaching it. Mike’s checks and Jeremy’s rubric are what that teaching looks like for one person.
The dissent: The Browser Company cofounder and CEO Josh Miller, who opened Thesis, thinks handing your work to AI is the wrong goal. “I don’t believe in this world of delegating to AI as the right relationship to this technology,” he told Laura. “I want to retain the agency in my craft.” His talk, which introduced his idea-sharpening voice app Clackety Clack, made the case from the other side: “What if AI prompts you so you can come up with the right answer?”
Steal these interview questions
When nobody can say what a job will look like in six months, what do you hire for? Curtis Anderson, founder and CEO of the nurse-staffing platform Nursa, shared onstage how he now starts every interview process with the same two questions:
- “Give me as many examples as you can… where you’ve been in an ambiguous situation and had to survive or grow through it.” The answers don’t have to come from work. Anderson will take “a physical struggle, a personal struggle, an emotional struggle.”
- “Tell me how many near-death experiences you’ve had.” He admits it’s “a little bit exaggerated.” What he really wants to know is how often you kept going “even when common sense or pain thresholds told you to stop.”
What he’s testing: “Do you have tolerance for uncertainty?” Nursa’s first value is that “execution is everything,” and as Anderson sees it, “so much of life is about iterations. You just need at-bats.” His example is Thomas Edison, who by the popular telling went through thousands of failed experiments before he got a working light bulb.
“Somewhere in those attempts it gets really discouraging, and discouragement keeps you from pushing the envelope.” So he hires the people who keep going anyway, the ones who are “actionably convicted.”
Try it: Ask question one in your next interview, or answer it yourself before your next review.
Discuss
“We like to watch other meatball[s] play chess, not silicon play chess.”—Ivan Zhao, cofounder and CEO of Notion, backstage at Thesis
Zhao’s idea starts with a book he’s been listening to: The Relentless Revolution, Joyce Appleby’s history of capitalism. As Zhao tells it, Appleby argues that the modern economy took off with a change in culture before a change in technology. Once people stopped thinking in zero-sum terms, farmers stopped growing only their own food and started to “buy each other’s things.”
Zhao thinks the same shift could happen with AI. Models produce “silicon tokens”; people produce what he calls “meatball tokens.” Even in a world with an abundance of silicon tokens, from models smarter than we are, he doesn’t think humans become worthless. We can “become consumers of other people’s human tokens,” he says. “We might like your expression, your poetry more than a language model’s expression.”
His evidence is sports. Computers have beaten the best human chess players since IBM’s Deep Blue beat Garry Kasparov in 1997, yet people still watch humans play. Cars outrun every sprinter, and we still watch the 100 meters.
Dan’s keynote made a similar point from the maker’s side: When the whole team ran the same customer survey through Claude, the value came from what each person chose to pull out, down to Jack Cheng’s Claymation game show. Zhao comes at it from the audience’s side: People will keep choosing some work because of who made it. He offers that as a hope more than a forecast: “I hope there’s hope there.”
Jack Cheng turned Every’s customer survey into a Claymation game show using Claude Opus 5.5, GPT Image, and video-generation models accessed through fal.ai.
Log on
Get hands-on with how Every uses AI at live camps, workshops, and meetups.
- How We’re Working Right Now: Claude Fable 5.2: Six of our seven testers gave Anthropic’s new frontier model a gold medal. It built Kieran Klaassen a working video editor in three days, but it’s slow and hungry for tokens. Read our Vibe Check (if you encountered a broken link in yesterday’s email, this one should work), watch Dan put it to work, or catch the replay of our livestream. Then join us Friday, November 13, noon to 1 p.m. ET on Zoom to see how we use it. Free for paid subscribers.
- Thesis: 2027: All 25 sessions from the Agora, Classroom, and Salon are available to watch now for paid members and will be released for free on our YouTube channel over the coming weeks. Watch the recordings.
Grant Harvey is a staff writer at Every. To read more essays like this, subscribe to Every and follow us on X at @every and on LinkedIn.
Special thanks to Eleanor Wikstrom for editorial support.
Stop explaining AI to your team and start showing them. The Every Agent does the work in Slack, where everyone can see it, so one good workflow quickly becomes the whole team’s.
