Our users often cite the redirect manager as their favorite tool in Yoast SEO Premium, but the internal linking tool is also high on the list. For months, we worked hard to improve the internal linking tool making it smarter and faster. I sat down with our developers Natalia Shitova and Hans-Christiaan Braun, two of the developers involved, to help give you the lowdown on this new internal linking tool and the magic that happens behind the scenes.
The internal linking tool in Yoast SEO Premium
For those not familiar with the internal linking tool, a short recap. This tool analyzes the content on your site to automatically give suggestions for links relevant to the post you are writing at the moment. It saves you a lot of time and maybe even your sanity. You don’t have to remember all those hundreds of posts you’ve written over the years. The internal linking tool surfaces them if relevant to the content you are writing now. After that, simply copy the link and paste it where you want it to appear. Easy peasy!
The initial version of the internal linking tool worked fine, but we saw a lot of potential enhancements. For one, we wanted to make it much faster. Plus, we wanted to make it more accurate and reliable. We noticed the tool didn’t always give the best answers. Sometimes, it’d miss a couple of posts or suggested less relevant ones. That’s just not an option for a tool people rely on, so we wanted to fix that.
The new version in Yoast SEO Premium 14.7 won’t look too different, but the way it works has totally changed — for the better!
Here’s the new internal linking tool in action:
Understanding content to suggest links
Internal linking is incredibly important and that’s one of the reasons we built the internal linking tool. For this to work well, the tool must know about all your content and understand it as well. The initial version of the internal linking tool was awesome, but it didn’t always give valuable suggestions. How come?
According to Natalia, looking for the most relevant posts to link to is a challenging task for both a human and a plugin. “You need to have a good understanding of what your current post is about. Plus, you need to keep a detailed overview of what all other posts on your website are about. In the new version of the internal linking tool, we attempted to make the analysis as intuitive as possible. We tried to make the plugin work in a way a human would — provided one with the perfect memory of a computer.”
How it was before
The initial version of the internal linking tool was one of the first WordPress SEO tools to automatically suggest relevant links based on your content. But how did that work?
Natalia explains: “The first version relied fully on topics of separate posts and did not take the site-wide context into consideration. The plugin would determine the most often used words and word combinations of all posts on the website — we call them “Prominent words” of the text — and then simply match those: the more words and word combinations matched between two texts, the more “similar” those texts were deemed and the better internal linking suggestion one was considered for the other.”
That sounds like a very logical process, but why wasn’t this approach optimal?
“The problem with this approach,” says Natalia, “is that it counts all words as equally valuable when it comes to determining the topic of the text, but that’s not the case.”
“Let’s take yoast.com as an example. Almost every post that we write is about SEO, WordPress, Yoast plugins, website optimization. Are these words important when we want to describe the topic of the post on its own? Of course! Are they worth looking at if we want to determine if two posts from yoast.com are related? No. To give better internal linking suggestions, we need to focus on often-used words but those that are more specific to a group of blog posts. In the case of yoast.com, it’s better to look at words like “copywriting”, “site structure”, “structured data”, etc. This is what we did in the new version of the internal linking tool.”
Old vs new: a quick comparison
A lot of the behind the scenes improvements are about understanding the content on your website better. We’ve optimized the way we collect words from posts. In the initial version, we used only the text of the article. The new version also looks at the title, meta description, taxonomies and eventually keyphrases. But the most important change is that we now consider the entire website as context when creating internal linking suggestions.
Natalia explains how that works: “When we try to determine the topic of a post not only do we check how often you use a certain word in this text, but also how often you use that word in other posts. These two measures together help us see how specific this word is for this post and whether it is valuable to use it for internal linking suggestions: a few words that you use a lot in the current post and rarely elsewhere will likely capture the specific topic of the post very well.”
Improving the internal linking tools started with research and building a demo to prove that it would work. Natalia: “For this demo, we took ten yoast.com texts strongly related to each other — all of them covered the topic of branding. We also selected ten control texts that were not specifically related to each other or to the branding texts from the first group. We ran the texts through both old and new internal linking algorithms and visualised how likely the algorithms would suggest every two texts to each other. As you see on the plot below, the old algorithm did not do a very good job. Strongly related texts (Text 1 through Text 10) were only considered slightly better linking suggestions for each other by the algorithm [red triangle], than unrelated texts (Text 11 through Text 20) [blue square]. The new algorithm definitely preferred linking strongly related texts over unrelated texts.”