Screaming Frog in Google Sheets

Screaming Frog is my favourite website development tool. It’s the best website crawler available at the cost, in my opinion, at least.

The tricky part of working with Screaming Frog (which you will have purchased separately) is just managing the files it produces, and prioritising that list in a time-efficient manner.

Hobo SF for Google Sheets (available to buy here) is one Google Sheet, Sheet Template, and App Script Library you have access to, to add to your Screaming Frog workflow to do this exact thing.

When set up to work in concert with a Scheduled Crawl in your previously purchased licensed version of Screaming Frog it will autonomously cycle through loaded clients, load Screaming Frog Files, manage your Google Drive and email you (or your clients) your reports.

During beta testing, I have not run into API costs. It works perfectly on a free Personal Gmail account (tested with about 150 clients) although will work best with a Google Workspace account).

There is no recording of data at the developer’s end (mine).

It is the real deal: a private, simple tidy, managing and reporting script for your managing SEO Audits and website development or migration projects and using Screaming Frog to its full reporting potential.

This uses your API credentials entirely through Google Sheets authorisation.

Key benefits

1. Time Saver – Identifies your website development priorities in minutes.
2. Time Saver – Manages, stores and collates 20-30 critical Screaming Frog crawl files saved to your Google Drive by Screaming Frog Scheduler (in minutes).
3. Time Saver – Tidies your Screaming Frog files in Google Drive, moving used files to the Google Drive trash for 30 days – so restorable, too.
4. Time Saver – Loads clients to a schedule you set.
5. Time Saver – Built-in checks to check you are sending clients the correct report.
6. Organiser – One report sheet is continually updated for the client report. Track your client data on one report sheet. The system creates a new document and continually updates that document for your client so it is one document you ever share with the client that is continuously updated.
7. Time Saver – Emails you and your clients (optional) an SEO Audit based on Screaming Frog crawl data of the site to a schedule you set.
8. Time Saver – Performs an automatic check against Screaming Frog data (which identifies content indexing issues) and a Hobo Premium SEO Checklist (guidelines laid down in Google’s documentation).
9. Secure – Automatic versioning Backup, Sharing, Access Privileges and Security built-in handled by Google.
10. Library Based (App Script).

Hobo SF for Google Sheets, working in concert with Screaming Frog, Google Sheets, Google Drive and Gmail helps you set up a quick and automatic SEO audit reporting system for multiple websites to a schedule you set.

 

Key features:

NOTE – I used Ai to generate a feature list directly from the code it analysed that is behind Hobo SF for Google Sheets, and here it is below:

Hobo SF (Multi-Site) for Google Sheets

Hobo SF for Google Sheets is a tool designed to automate SEO analysis using your Screaming Frog data. It operates directly within Google Drive and Sheets, streamlining the process of identifying SEO issues such as underperforming pages and broken links. This tool is particularly efficient, working within the runtime limits of a personal Gmail account and capable of handling unlimited client sites without incurring high API costs.

By importing a copy of your Screamingfrog crawl outputs, Hobo SF ensures a comprehensive review of your site’s SEO performance. Its automated reporting feature schedules and sends updates directly to you and your clients, saving significant time in manual reporting and analysis. Hobo SF emphasizes efficiency and cost-effectiveness, utilizing caching to minimize operational expenses and focusing on delivering direct, actionable insights to improve your site’s SEO.

It does this by:

1. Retrieving and Processing Specific Files from Google Drive – Finds and processes documents from Google Drive for SEO review.
2. Identifying Underperforming Pages – Highlights pages lacking engagement to signal performance issues.
3. Searching for New Missing Link Errors – Detects recent 404 errors for correction.
4. Finding New Server Error Instances – Identifies fresh server (500) errors affecting page display.
5. Coordinating SEO Analysis Process – Manages a thorough SEO audit for site visibility improvement.
6. Checking for External Broken Links – Documents broken inbound links from other websites.
7. Identifying Images Without Descriptions – Finds images missing alt text for SEO and accessibility.
8. Scanning for Potential Security Risks – Looks for site vulnerabilities compromising security or trust.
9. Looking for Pages Missing Main Headings – Searches for pages without H1 tags, essential for SEO.
10. Detecting Errors in Page Numbering Sequence – Corrects sequencing errors in page navigation.
11. Identifying Errors in Navigation Links – Fixes navigational link issues for better site usability.
12. Looking for Mixed Content Security Issues – Identifies secure/insecure content mix risking security.
13. Finding Pages Without Titles – Pinpoints pages missing titles, crucial for search engine listing.
14. Checking for Non-Searchable Reference Links – Ensures canonical links are searchable by engines.
15. Identifying URL Issues with Spaces – Detects space issues in URLs causing broken links.
16. Scanning for Unsecure Forms – Highlights forms with security risks.
17. Looking for Mobile Page Linking Issues – Identifies indexing issues with mobile page links.
18. Identifying Internal Links with No Response – Finds dead internal links affecting navigation.
19. Checking for Unsafe Links Between Websites – Ensures external links are secure.
20. Detecting Soft Error Links Within the Site – Identifies internal soft 404 errors.
21. Finding Pages with Access Blocked to Content – Searches for pages with blocked content access.
22. Identifying Redirects Within the Site – Looks for unnecessary internal redirects.
23. Searching for Redirect Chains Within the Site – Identifies and corrects redirect chains.
24. Looking for Non-Secure External Links – Checks external links for security issues.
25. Analyzing for Nearly Identical Content – Searches for duplicate content affecting SEO.
26. Identifying Pages Lacking Incoming Links – Finds pages without internal linking.
27. Inspecting for Pagination Issues – Checks for pagination errors affecting indexing.
28. Cleans and Organizes Google Drive – Manages space by moving used files to trash.
29. Adheres to Personal Gmail Account Limits – Ensures operation within Google’s script runtime limits.
30. No Limit on Client Sites Handling – Manages SEO tasks for any number of client sites.
31. Schedules and Emails Client Reports – Automates reporting to clients on a schedule.
32. Automatically Loads the Next Client – Moves to the next client seamlessly after task completion.
33. Runs Reports According to Schedule – Automates report generation and execution.
34. Uses the Cache at All Times – Reduces API costs by utilizing caching for data retrieval.

More details to follow.

The Screaming Frog Integration

This is a quick video demonstrating what the Screaming Frog integration does. If you are an experienced SEO or web developer you know organising and managing all these files can get pretty messy over time and managing them can be an art in itself.

Hobo SF is designed to speed all this up and take something that can take hours and do in it minutes.

 

Setting Up Screaming Frog

This system is designed to work with exported documents from Screaming Frog, and it should only take you moments to set this up.

When you purchase a Screaming Frog license, you can start to schedule crawls.

When you schedule a crawl, you want to save the exported files to your Google Drive. Screaming Frog Scheduler will format the files as they need to be for use in Hobo SF.

(Note – You can upload Screaming Frog files to your Google Drive manually, even using the free version of Screaming Frog for export files, but make sure you save the format as a Google Sheet specifically if you are hacking it like this. A rule of thumb is the Hobo SF files use GREEN files in your Drive (Google Sheets), not BLUE files, even if they are named the same, possibly with .csv extension. Just make sure your manual files are saved in Google Sheets format. )

I am currently grabbing screenshots for all this, I will update this later.

Important files to export to Google Drive from Screaming Frog will be the Crawl Overview, Internal_html, and any other files you are interested in.

A full list will be available here shortly.

Note once the Hobo Sf system uses a Screaming Frog export file, it moves it to the Drive trash for 30 days, so it won’t be used again in future crawls.

1. Set up a Scheduled Crawl in Screaming Frog

Name your task and Project name whatever you wish, and set up the frequency of your crawls.

1.2 Configure Start Options

Add your website URL and your config file location (you save this in Screaming Frog)

You can save a configuration file in Screaming Frog for any particular client.

1.3 Set Up API access

Set up API access in Screaming Frog (Hobo SF uses Google Search Console data for now, but will soon also include Analytics).

2. Choose export files

Set up export files in Screaming Frog that Hobo SF for Google Sheets uses.

Just copy all my settings in the images:

2.1 Choose export tabs

Set up export files in Screaming Frog that Hobo SF for Google Sheets uses.

At the moment Hobo SF uses two Export Tabs:

2.2 Choose Bulk Exports

Set up Bulk Export files in Screaming Frog that Hobo SF for Google Sheets uses.

Hobo SF 3.2 uses the following Bulk Export Reports:

and:

(I’ll clarify the above asap)

2.3 Choose Reports

Set up report files in Screaming Frog that Hobo SF for Google Sheets uses.

Hobo SF for Google SHeets uses the following reports:

And that’s it.

Make sure you save to Gsheet format and connect Screaming Frog to Google Drive.

Multi-Site Client Website Manager System

I’ve made it as simple and cost-effective as possible to manage an SEO audit process for multiple websites, to a schedule you have full control over.

The system is designed to manage multiple websites and multiple client website data files in your Google Drive folder.

Hobo SF for Google Sheets will run your SEO audits, manage your crawl files, create and develop a central client report, email clients links to the central report and load your next client to process.

You can use your own template with it (and add your own branding).

You can modify the Settings tab to hack reports and change limits on reporting and such.

Once it is set up it IS an autonomous reporter and will run to your schedule.

More Hobo SF Videos coming shortly:

1. Performing A Full Manual Audit with this system (Soon)
2. How it works underneath the hood so to speak (Soon)
3. Code Validation (Third-Party code validation – Soon)
4. All of Screaming Frog Reports (In the next major update)

See reactions for Hobo SF for Google Sheets on Linkedin and Twitter.

This is very much an indie-hacker project, and I’ve built a lot of it in public if you have been following along.

Not affiliated with Screaming Frog, and you will need to purchase a Screaming Frog license to use Hobo SF for Google Sheets.

As an SEO or web developer it will be the best purchases you’ve made recently.

NB: Note Current Version is V6. Any systematic abuse will not be tolerated. Your use is governed by a fair use policy and can be suspended (it is very unlikely normal use would trigger such an event, and I use it all day).