Programming

The Surprising Scale of the Small Web

Open WebRSSBloggingNews AggregationPlatform Decay
Added Mar 16
Link229
Article: PositiveCommunity: Very PositiveMixed

Author Kevin Boone investigated the feasibility of creating a feed aggregator for the 'small web' of non-commercial personal sites. His research revealed that the ecosystem has grown to tens of thousands of sites, with thousands of active contributors producing over 1,200 updates daily. This vibrancy proves that the private, ad-free internet is much larger and more resilient than many realize.

Key Points

  • The small web consists of personal, non-commercial sites that use standard web protocols but avoid corporate influence and tracking.
  • While the Gemini protocol community is small and manageable, the broader small web has grown significantly, with Kagi's index reaching 32,000 sites.
  • Analysis of RSS and Atom feeds shows that the small web produces over 1,200 new content updates daily across roughly 9,000 active sites.
  • The scale of the small web makes simple, all-encompassing feed aggregation difficult for a single reader to consume.
  • The growth of these sites represents a successful and healthy resistance to an internet dominated by commercial interests.

Sentiment

The Hacker News community is broadly enthusiastic about the existence and growth of the small web, validating the article's central thesis that it's bigger than commonly assumed. However, there's significant skepticism about current approaches to curating and defining it, particularly Kagi's restrictive criteria and the Gemini protocol's limitations. Many commenters feel that the real small web is orders of magnitude larger than what any list has captured. The discussion reveals a community that agrees with the diagnosis but debates the remedy.

In Agreement

  • The small web is indeed much larger than curated lists like Kagi suggest, with many commenters noting their personal sites aren't captured despite being quintessential small web
  • RSS and feed aggregators remain valuable tools for discovering and following small web content
  • Google's search algorithm has effectively buried the small web by optimizing for clicks and monetized content rather than quality
  • Alternative discovery tools like Marginalia Search, Kagi Small Web, and community-driven directories are important for surfacing this content
  • The small web represents a mindset of non-commercial sharing, not just a size category

Opposed

  • Kagi's criteria for the small web are too restrictive, excluding non-blogs, infrequently updated sites, non-English sites, and Substack-hosted content
  • The proliferation of exotic protocols like Gemini, Gopher, and Spartan is counterproductive — simple HTML without JavaScript would achieve the same goals with far less friction
  • Using niche protocols just creates insular technical communities that exclude non-technical people
  • The small web's anti-monetization stance may be self-defeating, as equating good with free cedes the field to corporations
  • The entire concept is fueled more by nostalgia than practical utility