If we would like to create a database of a set of websites and assign to these a popularity number this can become fairly complex fairly quickly.

1. root level domain and how to get second level domains

We start by creating a list of official all root level domains and official second level domains, for these we have as sources:

- http://www.iana.org/domains/root/db/ : the complete list of root level domains and, when clicking through, each registration organization in each country / part which gives us information on the official list.
- http://mxr.mozilla.org/mozilla-central/source/netwerk/dns/effective_tld_names.dat?raw=1 : an open source list on mozilla which gives a listing
- http://en.wikipedia.org/wiki/.XX : for each domain extension there are some parts on the official subdomains

In my experience until now none of these three individually provides a correct list. The Mozilla list is fairly outdated, the official registers often don’t specify the complete list of possibilities and here and there Wikipedia misses some. However when you put the effort in combining these you should end with a nice new overview. Which is what I did and which consists of hundreds of lines which I put in a regex to check it. E.g. for brazil:

        // —————————————————————-
        // .br (incl.ccTLD)
        // Brazil
        // http://registro.br/ / Registration date 1989-04-18.
        // notice: also br.com
        // http://en.wikipedia.org/wiki/.br
   ‘\.adm\.br|’ . // Administrators
   ‘\.adv\.br|’ . // Lawyers
   ‘\.agr\.br|’ . // Agriculture-related companies and/or websites
   ‘\.am\.br|’ . // AM radio companies, licenced by the Brazilian Communications Ministry
   ‘\.arq\.br|’ . // Architecture
   ‘\.art\.br|’ . // Art: music, folklore, etc.
   ‘\.ato\.br|’ . // Actors
   ‘\.b\.br|’ .  // * Exclusively for bancary environiment use (online banking, etc).
   ‘\.bio\.br|’ . // Biologists
   ‘\.blog\.br|’ . // Blogs
   ‘\.bmd\.br|’ . // Biomedics
   ‘\.can\.br|’ . // ?? mentioned on mozilla not in wikipedia
   ‘\.cim\.br|’ . // Realtors
   ‘\.cng\.br|’ . // Scenographers
   ‘\.cnt\.br|’ . // Accountants
   ‘\.com\.br|’ . // Commercial websites in general and individuals.
   ‘\.coop\.br|’ . // Cooperatives
   ‘\.ecn\.br|’ . // Economy
   ‘\.edu\.br|’ . // Higher education institutions
   ‘\.eng\.br|’ . // Engineers
   ‘\.esp\.br|’ . // Sports
   ‘\.etc\.br|’ . // Entities that do not fit in other categories
   ‘\.eti\.br|’ . // I.T. Specialists
   ‘\.far\.br|’ . // Pharmacies and drugstores
   ‘\.flog\.br|’ . // Photoblogs
   ‘\.fm\.br|’ . // FM radio companies, licenced by the Brazilian Communications Ministry
   ‘\.fnd\.br|’ . // Phonoaudiologists
   ‘\.fot\.br|’ . // Photographers
   ‘\.fst\.br|’ . // Physiotherapists
   ‘\.g12\.br|’ . // K12 education institutions
   ‘\.ggf\.br|’ . // Geographers
   ‘\.gov\.br|’ . // Government entities
   ‘\.imb\.br|’ . // Real estate
   ‘\.ind\.br|’ . // Industries
   ‘\.inf\.br|’ . // Media and information
   ‘\.jor\.br|’ . // Journalists
   ‘\.jus\.br|’ . // Brazilian Judiciary Branch of the State.
   ‘\.lel\.br|’ . // Auctioneers
   ‘\.mat\.br|’ . // Mathematicians and Statisticians
   ‘\.med\.br|’ . // Physicians
   ‘\.mil\.br|’ . // Brazilian Armed Forces
   ‘\.mus\.br|’ . // Musicians
   ‘\.net\.br|’ . // Commercial activities. Formerly it was for providers of physical communication means, legally habilitated for the prestation of public telecommunication services.
   ‘\.nom\.br|’ . // People
   ‘\.not\.br|’ . // Notaries
   ‘\.ntr\.br|’ . // Nutritionists
   ‘\.odo\.br|’ . // Dentists
   ‘\.org\.br|’ . // Non-profit non-governmental entities
   ‘\.ppg\.br|’ . // Publicity specialists and Marketeers
   ‘\.pro\.br|’ . // Teachers
   ‘\.psc\.br|’ . // Psychologists
   ‘\.psi\.br|’ . // Online service providers
   ‘\.qsl\.br|’ . // Radio amateurs
   ‘\.radio.br|’ . // * Entities that wish to broadcast radio
   ‘\.rec\.br|’ . // Recreational activities, entertainment, leisure, games, etc.
   ‘\.slg\.br|’ . // Sociologists
   ‘\.srv\.br|’ . // Services providers
   ‘\.taxi\.br|’ . // Taxis
   ‘\.teo\.br|’ . // Theologians
   ‘\.tmp\.br|’ . // Temporary events, such as fairs and exhibitions
   ‘\.trd\.br|’ . //  Translators
   ‘\.tur\.br|’ . // Tourism
   ‘\.tv\.br|’ . // Sound and image broadcasters, licenced by the Brazilian Communications Ministry
   ‘\.vet\.br|’ . // Veterinarians
   ‘\.vlog\.br|’ . // Videologs
   ‘\.wiki\.br|’ . // Wikis
   ‘\.zlg\.br|’ . // Zoologists

This is quite some work but it gives you a good feeling on the official ones and you will create a better class than classes which are merely based on the Mozilla listing.

Please remind yourself that you should check on the last 2 parts of an url and not to use a simple preg_match on the complete url for the second level domains since then you would also find e.g. http://whatever.co.uk.leau.net  which is obviously not correct.

2. unofficial "top" level domains

Although you may now think that your job is done since you have a nice list of all official domains, the truth is far from that. Because there are also kind-of unofficial popular top level domains.

For example: blogspot.com

Obviously you want to count each blogspot user’s blog as a unique thing with under this the blog entries the person wrote. (apart from blogspot.com kind-of sites there are also lots of "unofficial nic’s" operating under certain subdomains).

And there the trouble starts because:

a) each entitiy owning a domain and having multiple users or organizations under it uses a different structure for organizing it.

e.g.:

• for blogspot you would have myname.blogspot.com (or wordpress: lorelle.wordpress.com)
• for e.g. youtube you have youtube.com/user/whatever
• for sellaband users you have sellaband.com/projects/cecilemorel/

So in other words: where on the official IANA / TLD structure there is "something defined" that you can structure, on this level there are a gazillion possibilities, some use subdomains and some use paths to "a specific part".

And in the example we give url’s that can be followed in some way by url hacking but there are also sites which simply use index?r=45 and index?r=85 … People can really invent anything

b) each entitiy can be split in multiple parts having multiple different sets of different trees think e.g. about the live.com structure, so you would have to find a way to find a certain string but… the string must be in the correct place e.g. mycopy.of.blogspot.com.leau.net/~unique_user should not be counted for blogspot. ( probably by partitioning your database based on the official first and second level domains in the first place but… you will find the same problem deeper down the string e.g. whatever.youtube.blogspot.leau.net/user/whatever.

c) some parts under a domain would belong to the main domain and some parts would belong to a specific user / organization / project. So even if you define a certain url as a unofficial "root" it could well be that some parts under there actually belong to the parent and not the child.

3 IP addresses

Then… you would also have IP addresses that are used for more than 1 site and which give some headaches on how to count them.

4. More

I have only scratched the surface here so therefore "part 1" , stay tuned for part 2