Screaming Frog SEO Spider has been a tool we have been using for a few years now, which isn't as well known over here as it is in the UK and US markets.

Some of you may know why it's useful to have in your arsenal of SEO tools. For those of you who don't, Screaming Frog is basically a program which crawls URLs for a given website and gives back various on-site data such as page titles, meta descriptions, heading tags, and much more!

Screaming Frog

One of the latest features is the 'Custom Extraction' function, which allows you to put in your own CSS or Xpath in order to extract an HTML element from within a URL. This is useful for retrieving Google Analytics IDs, social media tags, product descriptions and more depending on what you want to do!

The video shows how to identify duplicate content in regards to product descriptions on an eCommerce website using this new feature on Screaming Frog.

Here is a quick summary of the steps from the video:

  1. Know what you want to extract - in our example it's the product descriptions.
  2. Right click on the webpage and 'Inspect Element' (or similar depending on your personal browser prefence).
  3. With the magnifying glass tool (top left), find what section you want to extract. This will navigate you to the area within the HTML which you've selected.
  4. Right click and copy 'CSS path' or 'Xpath'.
  5. Now go back to Screaming Frog and navigate onto Configuration > Custom > Extraction.
  6. Select a channel with your chosen method (CSS path, Xpath or Regex) and enter what you copied earlier from Inspect Element.
  7. Put in your chosen domain, URL or list depending on what you want to crawl.
  8. Make sure you navigate to the Custom > Filter (Extraction) and once you click start, it should reel in your chosen filter information if set up correctly.

Please note the element selected may not be present on every page (as a product description wouldn't be on the homepage for example).

Let us know how you find the video and if you are using the tool in any creative ways? We'd love to know!