Disclaimer: I'm not directly involved in any of this, though I've followed some general discussions.
1. The Web is much less diverse than you think it is, for two reasons.
2. Zipf/Power functions: the largest sites are a huge part of the Web. If people are spending (say) an hour online per day, and 40 minutes of that are FB/Insta, then all the rest of the Web gets 20 minutes. And much of that will be the next most popular site. Attention is fundamentally limited. In terms of experience, the Web is fairly small.
3. Which of course isn't content, but here again, much of the original content only hits a relatively small number of sites. I have looked into news site stats, and what you'll see for national papers of record (NYT, WSJ, WashPo, etc.) is about 150-500 items per day (Sundays are highest). The news services (AP, UPI, AFP, Reuters) tend to run 1,000 - 5,000 items/day. User-generated content sites are obviously higher than this, though much of that content is derivative (sourced/linked elsewhere).
4. Web engines. Even with a diversity of sites, there are a limited number of systems driving this. I've been noting a consolidation around news sites of a number of standard templates, and of course FB, YT, Instagram, Reddit, etc., will get you a bunch of specific Web engines. Which means it's less "one site per archiver" and more "what sites does this archiver work on?". And worse: over what period of time -- just because an archiver works on www.example.com from 2018-1-1 to 2018-12-31 doesn't mean it works from 2019-1-1 onward.
There are a few other approaches. Archive.today (archive.is / archive.fo) basically translate pages to their own representation. There are standardised presentation engines based on Readability (now defunct as a site, though its parser remains), used by Mozilla's Pocket and other tools (InstaPaper, etc.).
5. I also suspect we may see at least some normalisation around HTML5 (or subsequent) elements. Which I'd really like to have happen. NY Times, for example, has abandoned <table> for its own custom, and horribly broken, table-presentation alternative crap.
6. Yes, this leaves one-off SPAs which may be hard to archive. I don't know what the plans are for that.
(Now to see what IA have had to say about any of this.)
Update: from an IA response in this thread, "brozzler":
1. The Web is much less diverse than you think it is, for two reasons.
2. Zipf/Power functions: the largest sites are a huge part of the Web. If people are spending (say) an hour online per day, and 40 minutes of that are FB/Insta, then all the rest of the Web gets 20 minutes. And much of that will be the next most popular site. Attention is fundamentally limited. In terms of experience, the Web is fairly small.
3. Which of course isn't content, but here again, much of the original content only hits a relatively small number of sites. I have looked into news site stats, and what you'll see for national papers of record (NYT, WSJ, WashPo, etc.) is about 150-500 items per day (Sundays are highest). The news services (AP, UPI, AFP, Reuters) tend to run 1,000 - 5,000 items/day. User-generated content sites are obviously higher than this, though much of that content is derivative (sourced/linked elsewhere).
4. Web engines. Even with a diversity of sites, there are a limited number of systems driving this. I've been noting a consolidation around news sites of a number of standard templates, and of course FB, YT, Instagram, Reddit, etc., will get you a bunch of specific Web engines. Which means it's less "one site per archiver" and more "what sites does this archiver work on?". And worse: over what period of time -- just because an archiver works on www.example.com from 2018-1-1 to 2018-12-31 doesn't mean it works from 2019-1-1 onward.
There are a few other approaches. Archive.today (archive.is / archive.fo) basically translate pages to their own representation. There are standardised presentation engines based on Readability (now defunct as a site, though its parser remains), used by Mozilla's Pocket and other tools (InstaPaper, etc.).
5. I also suspect we may see at least some normalisation around HTML5 (or subsequent) elements. Which I'd really like to have happen. NY Times, for example, has abandoned <table> for its own custom, and horribly broken, table-presentation alternative crap.
6. Yes, this leaves one-off SPAs which may be hard to archive. I don't know what the plans are for that.
(Now to see what IA have had to say about any of this.)
Update: from an IA response in this thread, "brozzler":
https://github.com/internetarchive/brozzler