Context Window

The Case Against Skills

Use them intentionally, or they could make your AI worse

by Laura Entis

Was this newsletter forwarded to you? Sign up to get it in your inbox.

Signal

Skills could be making your AI worse

It might be time to examine your elaborate skill library. Skills are reusable packages of instructions—and sometimes examples and tools—that load whenever they’re relevant to what you asked your AI to do. They’re supposed to improve an agent’s performance. The format, popularized by Anthropic, is all over X, where sprawling custom skill libraries are treated as status symbols.

Mike Taylor, Every’s head of tech consulting, thinks every skill should earn its place in your library with proof it improves outcomes. His argument: Frontier models are smart enough that they’ve absorbed the need for most of the skills trending on social media. If a model can reason through something on its own—and Fable 5 or GPT-5.6 likely can—adding extra instructions creates confusion, not clarity. “You’re fighting the weights of the model by forcing it to do things your way instead of the way it was trained,” Mike says. Any time you conflict with the model’s training, it’s more likely to make mistakes. All the additional text loaded from your skill will also inflate costs, so you should make sure each skill you choose is worth it.

Skills are still useful, Mike says, but only when you need the model to complete a workflow in a specific way, like producing a custom PowerPoint template with instructions on your brand style guide or referencing internal company data. Mike recently found when testing Fable 5 that some of the skills Opus 4.8 needed to avoid mistakes actually harmed the newer model’s performance. It reminded Mike of his work as a prompt engineer in 2023: “With GPT-3 we had to use all these hacks and magic words to get it to produce valid code. Then when GPT-4 came out, it followed instructions better, and our bag of tricks was no longer necessary.”

Why it matters:

The data backs him up. SWE-Skills-Bench, a research benchmark that tests whether agent skills make agents better at software engineering, tested 49 public software-engineering skills and found that 39 didn’t impact performance, while three made things worse. At the same time, a large percentage of skills caused the model to consume more compute without improving results. (The worst offender increased token use by 451 percent.)

Only seven skills improved outcomes, and according to the researchers, these successful skills all provided specialized guidance the model couldn’t otherwise supply, like financial-risk formulas or traffic-management instructions.

What it means:

Skill utility has a shelf life. Instructions that patch a model’s blind spot can become redundant—or actively counterproductive—the moment a new version of the model absorbs that capability. The skills built to last are the ones that give the model information it couldn’t have known about your business or the way you work because it’s not public information: personal preferences about your writing style, a specific company template, internal company data, or an exact sequence of steps. When you do use skills made by other people, make sure they’re regularly updated and pruned by their author.

Try it this week: Perform a skills audit.

Keep skills that provide private context, custom tool access, personal taste, or a specific company workflow—things that people outside your company wouldn’t know.
Retest skills that compensate for a general weakness or quirk of a current model—these likely have a shelf life as the models improve.
Retire skills that don’t demonstrably improve results. You can ask your favorite AI agent to run your prompt with a skill and without, then compare the results.

Steal this workflow

Evaluate your skills to make sure you’re getting the results you want

“Writing lots of skills isn’t just productivity theater—you could be harming performance,” Mike says. “If you’re going to create a skill, prove that it works.”

Here’s his approach for doing just that:

Become a paid subscriber to Every to unlock this piece and learn about: