How to Run a Screaming Frog Content Audit & Rank Well

Content SEO audits are crucial for large websites, but they can feel like a challenge. Don’t worry – Screaming Frog SEO Spider is here to simplify the process.

This tool isn’t just another gadget in your SEO toolbox — it’s a content audit powerhouse. From uncovering broken links to analyzing meta descriptions and uncovering duplicate content, Screaming Frog SEO Spider is indispensable for SEOs and website owners alike.

After years of use, I can confidently say that it’s invaluable for this task. Find out why mastering this tool could be the game-changer your digital presence has been waiting for.

Table Of Contents:

• Why You Need a Screaming Frog Content Audit
• Setting Up Your Screaming Frog Content Audit
• Running Your Screaming Frog Content Audit
• Advanced Screaming Frog Content Audit Techniques
• Making Sense of Your Screaming Frog Content Audit Data
• Implementing Changes Based on Your Audit
• FAQs: Screaming Frog Content Audits
• Conclusion

Why You Need a Screaming Frog Content Audit

Before we explore the how-to’s of using Screaming Frog SEO Spider, let’s understand why these audits are so important.

SEO audits help you uncover hidden issues within your website’s content, structure, and SEO elements. Consider it a comprehensive check-up for your site where Screaming Frog gathers valuable SEO data.

Regular Screaming Frog content audits enable you to:

• Find and fix broken links.
• Identify instances of duplicate content.
• Optimize your meta descriptions and titles.
• Improve the structure of your internal linking.
• Discover opportunities for enhancing your website content audit and content quality.

Now that you understand the importance of a site audit, let’s take a closer look at how to perform a Screaming Frog content SEO audit.

Setting Up Your Screaming Frog Content Audit

First and foremost, you’ll need to download and install the Screaming Frog SEO Spider app.

A free version is available for crawling up to 500 URLs, which might be sufficient for smaller websites.

However, for larger sites, investing in the paid version is recommended.

Configuring Your Crawl

Before initiating your audit, it’s crucial to configure your crawl settings. Here’s what I suggest:

1. Navigate to “Configuration” > “Spider”.
2. Under the “Basics” tab, ensure that “Follow Internal Nofollow” is checked. This will allow Screaming Frog to discover orphaned content that might be excluded from your XML sitemaps.
3. In the “Extraction” tab, select all the elements you intend to analyze, such as titles, meta descriptions, and H1 tags. This data will help you analyze common SEO issues.
4. If you’re auditing a large website, think about adjusting the “Speed” settings to avoid overloading the server. You can even specify crawl depth if you only want to audit a specific section of your website.

Connecting APIs for More Data

To enhance your audit, consider connecting relevant APIs. This step, while optional, is highly recommended. You can easily audit redirects and find issues in redirect chains as well.

Here’s how to do it:

1. Go to “Configuration” > “API Access”.
2. Connect Google Analytics 4 and Google Search Console for traffic and search data. You can even integrate Google Analytics directly into your Screaming Frog reports.
3. If you have access, connect Ahrefs or Moz for backlink data. This data is useful for understanding your website’s backlink profile and identifying any potential issues, including toxic backlinks or unnatural link patterns.

Integrating these APIs provides a more comprehensive view of your website’s performance. It’s like adding an extra layer of valuable data analysis for an even more effective audit.

Running Your Screaming Frog Content Audit

With everything configured, it’s time to start the audit.

Enter your website’s URL in the designated field and hit “Start.”

Whether your website uses HTML web pages or a different format, Screaming Frog is designed to handle it all, including broken links, and provides you with the exact duplicate content to review.

As the crawl progresses, you’ll see data populating in real time. It’s exciting to observe, especially for data enthusiasts.

Once the crawl is finished, it’s time to delve into the data. Below are the key areas to focus on during your audit.

1. Identifying Technical SEO Issues

Begin by checking the “Response Codes” tab. This will reveal any 404 errors, redirects, or other issues that require attention. You can then easily audit redirects within this area of Screaming Frog.

2. Evaluating On-Page SEO Elements

Next, examine the “Page Titles” and “Meta Descriptions” tabs. Ensure they are the optimal length and contain relevant keywords. Additionally, check for any duplicate titles or meta descriptions.

3. Assessing Content Quality

The “Content” tab is where the real value of a Screaming Frog content audit shines through. It displays word counts, highlights duplicate content, and even provides readability scores if you’ve enabled that feature.

Screaming Frog’s extraction feature can identify low-word-count pages, which may indicate thin content.

4. Analyzing Internal Linking

Explore the “Inlinks” tab to see how your pages are interconnected. This helps identify opportunities for improving site structure and effectively spreading link equity.

Advanced Screaming Frog Content Audit Techniques

Once you’re comfortable with the basics, consider some advanced techniques to elevate your site audits.

Custom Extraction

A particularly useful feature is custom extraction, enabling you to pull specific data points from your pages that aren’t part of the default crawl.  The custom extraction feature utilizes XPath, CSS selectors, and regex to help with technical SEO tasks, making Screaming Frog an even more valuable SEO spider tool for SEO auditing.

For example, you could extract author names, publication dates, or any custom fields used on your site.

To configure this:

1. Go to “Configuration” > “Custom” > “Extraction”.
2. Click “Add”.
3. Select your preferred extraction method from CSS Path, XPath, or Regex.
4. Enter your selector and assign it a name.

Content Grouping

Content grouping is another powerful technique. Screaming Frog enables you to segment and analyze different sections of your website separately. You can use this information to make informed decisions about your content strategy and to improve the user experience on your site.

For instance, you might want to differentiate between your blog posts and product pages.

To group content:

1. Go to “Configuration” > “Custom” > “Search”.
2. Define rules for content categorization. For instance, URLs containing “/blog” can be grouped as “Blog Posts.”

Content grouping significantly streamlines your audit, particularly for larger sites with diverse content types.

Crawl Depth Analysis

Ever feel like some of your website pages are lost in the digital abyss?  That’s where crawl depth comes in. Crawl depth, also known as link depth, is a critical SEO factor that measures the number of clicks it takes for a search engine bot to reach a specific page on your website from the homepage.  A lower crawl depth generally means better visibility to search engines.

Think of it this way: the homepage is level 0. A page linked directly from the homepage has a crawl depth of 1, a page linked from there has a crawl depth of 2, and so on. The further away a page is from the homepage, the deeper it is buried in your site architecture, and the harder it might be for search engines to find and index it.

Why does this matter? Search engine bots have a “crawl budget,” meaning they allocate a certain amount of time and resources to crawling each website.  Pages with a high crawl depth might not get crawled as frequently, leading to lower visibility and potentially impacting your SEO performance.

So, how can you uncover these hidden gems and bring them back to the surface? One word: Screaming Frog. This powerful SEO tool helps you analyze your website’s architecture, including identifying pages with a high crawl depth.

Here’s how to perform a crawl depth analysis with Screaming Frog:

1. Enter your website’s URL and start the crawl.
2. Once the crawl is complete, navigate to the “Internal” tab.
3. Sort the “Crawl Depth” column from high to low.

This will show you all the pages on your website organized by their crawl depth. Now, pay close attention to important pages buried deep within your site structure. Consider these strategies to make them more accessible:

• Add them to your main navigation if they’re key pages.
• Internally link to them from relevant, higher-level content.
• Review and update your XML sitemap to ensure all important pages are included.

By understanding and optimizing your website’s crawl depth, you can improve your overall SEO performance and ensure all your valuable content gets the visibility it deserves.

Find Poorly Linked Pages

Poorly linked pages are a common problem that can significantly impact your website’s SEO. Even if you regularly publish high-quality content, if your internal linking structure is weak, your newer pages may struggle to gain traction and attract organic traffic.

Think of it this way: you have a website selling handmade jewelry, and you diligently blog about different jewelry-making techniques, showcasing your products along the way. Over time, some of your products become bestsellers, naturally accumulating more internal links from your blog posts. However, as you expand your product line, newer items might not receive the same attention, leading to a disparity in internal linking.

This is where the Screaming Frog SEO Spider swoops in. Here’s how it helps uncover poorly linked pages:

1. Crawl your website using Screaming Frog.
2. Navigate to the “Link” tab within the Screaming Frog interface.
3. Locate the “Unique Inlinks” column. This column displays the number of unique internal links pointing to each URL on your website. A “unique” inlink means that multiple links from the same page are counted only once.
4. Sort the “Unique Inlinks” column in ascending order to identify pages with the fewest internal links.
5. Alternatively, you can use the “% of Total” column, which shows the percentage of unique internal inlinks to each URL compared to the total number of unique internal links on your website. Pages with a low percentage may require attention.

By analyzing the “Unique Inlinks” data, you can quickly pinpoint pages that need some internal linking love. The next step is to find relevant and contextual opportunities to link to these pages from other parts of your website.

For a more in-depth understanding of internal linking and its impact on SEO, check out this comprehensive guide on Internal Links for SEO.

Remember, a robust internal linking structure not only improves your website’s crawlability and indexability but also helps distribute link equity, boosting the overall authority and ranking potential of your pages.

Find Orphaned Pages

An orphaned page is a web page that’s full of potentially valuable content but no internal links lead to it.  This means your website visitors, and search engine crawlers, won’t easily find it.

Think about it: if visitors can’t easily access a page, they probably won’t. And if search engines like Google can’t crawl a page, they’re less likely to index it, let alone rank it well in search results. This means you’re missing out on potential organic traffic and conversions.

Orphaned pages often crop up unintentionally. Here are some common culprits:

• Deleted Links: You might have important pages where incoming links were accidentally deleted or removed during a website redesign.
• Outdated Content: Old campaign landing pages, product pages for discontinued items, or outdated blog posts can become orphaned over time.
• Test Pages: Sometimes, test pages used during website development are left behind and forgotten.

With Screaming Frog SEO Spider, you dont’ have to manually click through every nook and cranny of your website to find orphan pages.

Screaming Frog crawls your website like a search engine bot, but it can also integrate data from sources like Google Analytics, Google Search Console, and your XML sitemap. This cross-referencing helps pinpoint pages that exist in these analytics and indexing sources but lack internal links within your website’s structure.

Once you’ve identified your orphaned pages, you have a few options. You can:

• Delete irrelevant or outdated pages.
• Implement 301 redirects to guide users and search engines to relevant, active pages.
• Revamp and update orphaned content, then integrate it back into your website’s internal linking structure.

Audit Core Web Vitals

A Screaming Frog SEO spider crawl can do more than just uncover technical SEO issues. When combined with PageSpeed Insights API integration, it transforms into a powerful tool for measuring Core Web Vitals. This allows you to analyze the page experience of your entire website and identify areas for improvement.

What are Core Web Vitals?

Core Web Vitals are a set of website performance metrics Google uses to measure the user experience of a webpage. They provide insights into how users experience a page’s loading speed, interactivity, and visual stability. Google uses this data as part of its overall page experience signals to rank websites in search results.

Currently, there are three main Core Web Vitals metrics:

1. Largest Contentful Paint (LCP): Measures how quickly the main content of a page loads.
2. First Input Delay (FID): Measures how quickly a page responds to user interactions, such as clicking a button or scrolling.
3. Cumulative Layout Shift (CLS): Measures the visual stability of a page and how much elements shift unexpectedly during loading.

Follow these steps to run a Core Web Vitals audit with Screaming Frog:

1. Get a PageSpeed Insights API Key: Sign up for a free API key on the Google Developers Console. This key enables Screaming Frog to fetch Core Web Vitals data directly from Google.
2. Configure Screaming Frog: Navigate to ‘Configuration’ > ‘API Access’ > ‘PageSpeed Insights’ within Screaming Frog. Input your API key and select your preferred settings, such as mobile or desktop data.
3. Start Your Crawl: Input the URL you wish to crawl and initiate the process. As Screaming Frog crawls your website, it will also collect Core Web Vitals metrics for each page.
4. Analyze the Results: Once the crawl is complete, you can view Core Web Vitals data within dedicated columns. Filter and sort this data to pinpoint pages needing improvement.

Optimizing each Core Web Vital requires a targeted approach. Here are some general tips to get you started:

Optimize Your Images: Large image files are one of the most common causes of slow loading times. You can optimize your images by compressing them, using modern image formats like WebP, and ensuring they are responsive.

Minimize JavaScript and CSS: Large JavaScript and CSS files can also slow down your page load times. Minifying these files removes unnecessary characters, reducing their size and improving load times. You can also defer the loading of non-critical JavaScript files until after the page content has loaded.

Use a Content Delivery Network (CDN): A CDN is a network of servers that deliver your website’s content to users from a server that is geographically closer to them. This can significantly improve page load times, especially for users who are located far away from your web server.

Making Sense of Your Screaming Frog Content Audit Data

With a wealth of data collected, it’s essential to know how to use it effectively. Here’s my process for transforming site audit data into actionable insights:

1. Export the data to a spreadsheet program like Excel or Google Sheets.
2. Use conditional formatting to highlight any issues. Examples include titles exceeding 60 characters and thin content under 300 words.
3. Create pivot tables to summarize data based on content groups or page types. Screaming Frog allows bulk export of data in various formats, such as CSV, for further analysis in tools like Excel.
4. Prioritize identified issues by their potential impact and the effort required to fix them. This approach ensures you’re addressing the most critical areas first.

Keep in mind that SEO audits are about finding problems and opportunities.  Look for high-performing pages and determine if their success can be replicated across your website.

Implementing Changes Based on Your Audit

So you ran a complete Screaming Frog content audit and found the culprits holding your website back. Now what?

It’s time to take action. Use the following checklist to guide your next steps:

• Fix any encountered broken links or 404 errors. These errors can negatively impact your site’s SEO and user experience.
• Optimize any underperforming page titles and meta descriptions to improve click-through rates from search engine results.
• Improve or consolidate thin content. Use Screaming Frog to identify these pages and decide whether to improve them or implement redirects.
• Enhance internal linking wherever suitable. Internal linking helps users and search engines navigate your site more easily. Screaming Frog’s features are designed to enhance your understanding of internal linking and make improvements.
• Resolve any duplicate content issues discovered during the audit. Screaming Frog can help you locate and manage these URLs, preventing them from competing with each other in search results.
• Keep all information up to date. Outdated content can hurt your site’s credibility and search rankings.

Need to search for new keywords? Improve thin content? Update old information? BrandWell can help you revamp your entire content strategy based on your audit findings.

Unearth new keywords and build topic clusters by simply entering a seed topic:

And then generate a 100% original, research-backed, long-form blog post that’s ready to publish:

If you need to repurpose old content, BrandWell gives you six options to recycle content:

Running a content audit with tools like Screaming Frog can uncover a wealth of insights about your website’s performance and SEO health. However, the true value lies not just in the analysis but in the actions you take afterward. Implementing improvements — whether it’s optimizing meta tags, fixing broken links, or refreshing outdated content — is crucial for realizing tangible benefits.

FAQs: Screaming Frog Content Audits

How to do an SEO audit with Screaming Frog?

Conducting an SEO audit using Screaming Frog starts with configuring your crawl settings and then initiating a full website crawl. The focus should be on analyzing results, paying attention to aspects such as:

• Page titles
• Meta descriptions
• Content quality
• Internal linking structure

Use the data to identify and prioritize any SEO issues, and uncover areas for potential improvement. You can create custom extractions and conduct custom searches based on your SEO strategy.

What information can you get from using Screaming Frog?

Screaming Frog offers a wealth of data, including:

• Page titles
• Meta descriptions
• Headings
• Word counts
• Internal and external links
• Response codes
• Redirect chains
• Canonical tags
• Hreflang tags

Through API integration, you can even gather insights on:

• Search analytics
• Backlink data
• PageSpeed insights

Is Screaming Frog free for commercial use?

Screaming Frog provides a free version allowing crawls of up to 500 URLs, suitable for smaller websites. However, for larger commercial websites, purchasing a license is necessary. The paid version removes URL limitations and grants access to advanced features, including:

• JavaScript rendering
• Google Analytics integration

What is a Screaming Frog report?

A Screaming Frog report delivers a comprehensive overview of your website’s technical SEO aspects. This includes data on all crawled URLs, encompassing:

• Status codes
• On-page elements
• Potential issues

You can customize these reports to zero in on specific areas of your site, making them highly valuable for SEO audits and ongoing site maintenance.

Conclusion

Screaming Frog content audits are indispensable for any SEO professional. They offer invaluable insight into your website’s overall health and performance, highlighting problems and revealing opportunities.

Regularly conducting these audits and addressing the findings helps keep your site optimized and ahead of the curve in the ever-evolving digital landscape.

Remember, successfully using Screaming Frog extends beyond just running the tool— it lies in your ability to interpret and leverage the data effectively.

By proactively addressing issues and strategically enhancing your content with an AI tool like BrandWell, you not only boost your SEO rankings but also enhance user experience, driving meaningful engagement and conversions. Remember, the audit is just the beginning; real growth comes from acting on its data.