Your Swift Tests Are Great. Until a Mutant Shows Up
Mutation score in Swift measures what code coverage can’t.
Don’t worry zombie apocalypse hasn’t happen yet🧟♂️
But after reading this article, you’ll learn a new metric for assessing the real quality of your unit tests ✨
I’ve worked on teams where code coverage was considered one of the most important metrics. And yet, even as coverage increased, the codebase didn’t necessarily become healthier 🏥.
In this article, I’ll introduce a metric that measures something code coverage doesn’t… 👇
Code-Coverage 🔎
Every’s developers nightmare is to ship a bug in production. A strong CI pipeline and a reliable test suite builds confidence and speeds up deliveries 🚀
And reliable is the key word here. 💵
A good friend and coworker once told me:
If you want to introduce something to the team, it needs to be measurable. 🕊️
So that raises an important question:
How do you measure whether your tests are actually good or bad? 🤔
Do you just wait until customers start complaining about bugs they find?
Code coverage may look great on paper…but reliability is what actually protects production.
There has to be a better way…
The problem becomes even worse if your team is scaling up. How do you know that tests written are reliable? And then… you end up with having many regression bugs although you’ve hit your code-coverage team goal 😱
Mutation Testing 🧪
💡Mutation testing is not a mobile-only concept.
After reading this article, ask your backend, web friends if they use it.
Konsta what is this about? Is this just a fancy name? Give me an example.
Sure, and let me explain, in order to do this type of test you need a mutant.
🐛A mutant is a change (a bug) that you create in your codebase by changing your production code.
What kinds of mutations exist?
Mutants are usually small, realistic changes such as👇
swapping operators (>= → <)
flipping logical expressions (&& → ||)
replacing constants (true → false)
deleting function calls, especially when return values are discarded.
Once the mutant is inserted, we run the test suite.
If test suite failed it’s a good sign and it means that mutant was eliminated (this increases the mutation score.) This happens when there is a test case that is testing against the age < 18.
If tests succeeded means that mutant survived and this lowers the mutation score. (which implies there was no test that captured the bug/mutant we introduced above)
ℹ️ The mutation score is:
eliminated mutants / total mutants
A mutation score of ~0% in code-covered areas probably means the tests are… 🗑️😅
Now, we’ve described the process for a single mutation. But how do we generate many mutants automatically and summarize the results in a clean report? 👇
Muter CLI 🧟
Automated mutation testing for Swift 👇
https://github.com/muter-mutation-testing/muter
It’s really easy for anyone to use:
Install the CLI via homebrew 🍺
Provide a muter.conf.yml (this tells Muter how to run your tests)
Run it on your most important files and see how well tested they are 🤞
Muter gives a detailed report that you can use to understand the weak spots on your test.
💡This can also be applied on the CI level to run on changed files. This can give valuable feedback to the reviewer/author. This is possible with the --files-to-mutate flag.
Practical Advice 💪
One important thing to keep in mind. Mutation testing can become slow depending on how many files you mutate.
So I recommend starting small ✅ run it selectively on files with heavy business logic first instead of trying to mutate your entire codebase from first run.
Last thoughts 💭
Mutation testing is still not very popular in the Swift community, especially compared to ecosystems like JavaScript..
But I genuinely believe it provides that “extra something” an additional layer of confidence when writing tests (whether you write them yourself or with a little help from AI 😅).
🙏I hope in the future Xcode support more customization so that mutation score was an option on Xcode to show directly in editor as codecoverage is doing already.
🎉Once you start thinking in terms of “would my tests actually catch a bug?” instead of just “did this line execute?”, it becomes clear why mutation testing is such a powerful addition to your testing toolbox.
Looking ahead to hear your thoughts and ideas
See you in the next article 🚀
Konstantinos Nikoloutsos