Web rot is rising

Traffic to top websites has fallen by more than 11 percent in the past five years, according to data from Similarweb—a clear sign of the challenges traditional publishers face in the Artificial Intelligence (AI) era. Internet usage and adoption continue to grow, but older websites are struggling to keep up as newer AI-driven experiences start to dominate user attention. Data shows that those older sites don't just magically disappear. They continue to rot on the open web for years, clouding search results and leaving behind trails of broken or outdated links. The slow downfall of older websites has left the internet muddled for users and scrapers trying to gather data to accurately train large language models. While some publishers have pushed to maintain and update their web experiences, many have instead neglected their old websites. A study from the Pew Research Center found that a quarter of all webpages that existed at one point between 2013 and 2023 are no longer accessible.