Duplicates of site pages may be complete or incomplete. The first ones completely repeat the canonical pages, the second ones do it partially (descriptions of products, other blocks of text).

There are several options for finding them. It's easier to find complete duplicates. To identify partial ones, you will have to spend more time and effort. This process cannot be automated.

Duplicate pages have a slight impact on ranking in Yandex. But Google is more critical of them. It can significantly lower your position in search results.

Google Webmaster Panel

In the Google WebMaster panel, select the “Optimization” item, and in it – “HTML Optimization”. In the resulting table, you need to pay attention to 2 points:
  • repetitive meta descriptions;
  • repeating titles.

Clicking on each of them opens access to duplicate pages. After all non-canonical duplicates have been found, you can delete them (if they are not needed), provide a link to the canonical address, or use 301 redirect for gluing mirrors.

Pages with duplicate content should be saved when they are created specifically - for example, for a mobile version of the site or a print version. The text may be repeated - for example, on the main page and in product categories (often in online stores ). In this case, it is better to rewrite one of the options.

Via search bar

In the search line you must enter the construction “site:your_site_address -site:your_site_address/&”. It will reflect all site pages from the general index, with the exception of those in the main index. This design allows you to identify:
  • uninformative pages;
  • partial duplicates.

They are considered by the system as unnecessary, uninformative or spammy, which lowers the overall position of the site in search results. The “repeat search with missing results” item provides an opportunity to see a more holistic picture of what is happening.

In the Yandex search engine, partial duplicates can be searched for in individual parts of the text. It shows all the results within the site, allowing you to eliminate problem areas.

Xenu

A special program for resource optimization is another option for searching for duplicates. It may turn out to be more effective. This is relevant if the site exists relatively recently, and not all canonical or duplicate pages are included in the index. The program analyzes the site independently. It displays duplicate content and reduces the time it takes to fix the problem.

Why do duplicates occur?

There are several reasons why duplicates appear:
  • features of the site engine. Many CMS duplicate pages under other addresses to solve certain problems. Existing plugins and tags provide a way to get rid of this problem;
  • inexperience of the webmaster. This is especially true on large sites. It will not be possible to identify and eliminate all errors immediately. Periodic checking and optimization will reduce the likelihood of their occurrence;
  • poorly thought out code. Lack of redirects, meta tags and incorrect operation of 404 pages are a common reason for the appearance of duplicates;
  • a lot of the same content. This problem is critical for large online stores with a large number of identical products. To prevent the search robot from perceiving descriptions as spam, they can be displayed via Ajax or iFrame.

What problems can duplicate pages cause:

  • rating downgrade due to spam;
  • dispersion of link weight. Natural links from users to duplicate pages have less effect;
  • filter overlay ;
  • loss of internal weight.

Eliminating duplicate pages is an important task that should be performed periodically.