🌍 Where Are DEV Users Coming From?

As you may already know, dev.to has an API to fetch articles and users.

Let's give it a try with my favorite HTTP client, httpie:

http https://dev.to/api/users/1

{
    "github_username": "benhalpern",
    "id": 1,
    "joined_at": "Dec 27, 2015",
    "location": "Brooklyn, NY",
    "name": "Ben Halpern",
    "profile_image": "https://res.cloudinary.com/practicaldev/image/fetch/s--DOU9qJSH--/c_fill,f_auto,fl_progressive,h_320,q_auto,w_320/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/1/f451a206-11c8-4e3d-8936-143d0a7e65bb.png",
    "summary": "A Canadian software developer who thinks he’s funny.",
    "twitter_username": "bendhalpern",
    "type_of": "user",
    "username": "ben",
    "website_url": "http://benhalpern.com"
}

What I first noticed is that id seems to be auto-incremented.
After some tries, I quickly found out the total number of users on dev.to: around 119000.
It gives me the idea to scrape all the users public data and try to play with it.

Let's try with this command:

for i in {1..119000}; do http https://dev.to/api/users/$i >> users.dev.to.json && echo "," >> users.dev.to.json && echo "https://dev.to/api/users/$i"; done

After a few hours running in the background, I finally got all the users in my JSON file.

I ended up by importing the JSON file in a mongodb collection using NoSQLBooster.

There isn't many information in the users' profile, but two of them worth having a look at them: joined_at and location. It could be very interesting to have a diagram showing the progression of registrations through time. Maybe I'll do it some day, but for now, let's compute some statistics on users' location.

I used two queries to compute the statistics. This one helped me to get the big trends in people's location:

db.getCollection("users.dev.to").aggregate(
    {"$group": {_id: "$location", count:{$sum:1}}}
).sort("-count")

Then, I use another query to get the count of people per location:

db.getCollection("users.dev.to").find({"location": /\bdublin\b/i})

The conclusions are the following:

Most of the users (90%) don't fill their location.
16.3% of the users who filled their location are from USA.
5.8% are from India.
4.9% are from UK.
4% are from Japan.
3.8% are from Germany.

Let's have a look at the full statistics below:

#	country	region or city	#region or city	#country	%
1	USA			1850	16.32%
		Incl New York	205
		Incl CA	194
		Incl NY	147
		Incl San Francisco	137
		Incl TX	119
		Incl Seattle	117
		Incl Los Angeles	107
		Incl Chicago	101
		Incl Washington	81
		Incl FL	78
		Incl PA	68
		Incl MA	67
		Incl AZ	38
		Incl MI	34
		Incl OH	30
2	India			660	5.82%
		Incl Bangalore	107
		Incl Delhi	103
3	UK			551	4.86%
		Incl London	258
		Incl England	35
		Incl Scotland	33
		Incl Wales	7
4	Japan			451	3.98%
		Incl Tokyo	242
5	Germany			428	3.77%
		Incl Berlin	128
6	Brazil			301	2.65%
7	France			294	2.59%
		Incl Paris	98
8	Canada			257	2.27%
		Incl Toronto	107
9	Nigeria			187	1.65%
		Incl Lagos	97
10	Netherlands			154	1.36%
		Incl Amsterdam	62
11	Spain			110	0.97%
12	Argentina			109	0.96%
13	Italy			102	0.90%
14	Indonesia			92	0.81%
15	Russia			89	0.78%
16	Mexico			86	0.76%
17	Philippines			85	0.75%
18	Poland			81	0.71%
19	Ukraine			75	0.66%
20	Portugal			63	0.56%
21	South Africa			57	0.50%
22	Pakistan			56	0.49%
23	Turkey			55	0.49%
24	Belgium			53	0.47%
25	Egypt			52	0.46%
26	Bangladesh			50	0.44%
27	Colombia			46	0.41%
28	Romania			45	0.40%
29	Kenya			45	0.40%
30	Switzerland			45	0.40%
31	Austria			43	0.38%
32	China			40	0.35%
33	Norway			38	0.34%
34	Ireland			33	0.29%
	Total main locations (listed above)			6683	58.94%
	Other locations (unlisted)			4656	41.06%
	Total locations			11339	100.00%
	<null>			90260	80.80%
	<empty>			10108	9.05%
	Total			111707	100%

Note: these numbers are one month old, but they give a good overview of the trends.

Thanks for reading!

Comments

[Ben Halpern]

Cool!!

[Yaser Al-Najjar]

I guess you might want to add throttling on the API based on IP ;)

[Ben Halpern]

Yes, but it’s also served from our CDN so it’s not a resource-gobbling endpoint.

[Boris Jamot ✊ /]

😱

[Matteo Joliveau]

Although cool in and of itself, this also highlights why using autoincremented ids is not a good idea when it comes to protecting users privacy (this is not the case of course since I don't see any critical data in the JSON response).

I always use UUIDs for exactly that reason, and also because it helps when building distributed systems (id clashing is less probable with them)

[Boris Jamot ✊ /]

I'm sorry, my english level is quite low. Should I have written "scrape" ?

[Andrew Bone]

I wouldn't say your English is bad, your writing is better than a lot of native speakers 😉

[Stefan Đokić]

At the time of posting this, DEV has 119,105 registered users. This is so cool 😁
dev.to/api/users/119105

[Noman Gul]

yeah => dev.to/api/users/254188

[Stephen Chiang]

What about me? I'm in Norway...says so on my profile :P

[Boris Jamot ✊ /]

Norway: 33 users.

[Boris Jamot ✊ /]

38 with Oslo.

[Stephen Chiang]

i'm sure there's quite a few from Denmark as well :)

[Boris Jamot ✊ /]

Sure, but there are a lot of other countries I didn't put in the list because their count of users is low.

[Stephen Chiang]

Ah I see, I just assumed it was a complete list because of this line:

Let's have a look at the full statistics below:

And to be fair 38 is more than Ireland ;)

[Boris Jamot ✊ /]

In fact, Norway and Denmark are in "other locations".

[Cosmin Popescu]

I am from Romania. :D

[Boris Jamot ✊ /]

Romania: 45 (Incl Bucharest)

[Michiel Hendriks]

Why?
I've been on the internet since 1997. I have never bothered about location, age, sex, species. (Although secretly I would be excited to converse with a cat.)

[Dan Lugo]

1 of 57 from South Africa!

[xelaflash]

Nice, it shows that this is a small world!
I'm 🇫🇷, but living in Barbados 🇧🇧, i would be interested in knowing if i'm the only one from there? 🤔

[Boris Jamot ✊ /]

db.getCollection("users.dev.to").find({"location": /\bbarbados\b/i})

{
    "_id" : ObjectId("5bdb776102a47d33d338dcad"),
    "type_of" : "user",
    "id" : 46999,
    "username" : "xelaflash",
    "name" : "xelaflash",
    "summary" : "I'm Alex, i've learned coding in 2018, i'm now willing to start my freelance career.\r\nHere to learn and grow... and learn but mostly learn 😁",
    "twitter_username" : "GhostXela",
    "github_username" : "Xelaflash",
    "website_url" : "",
    "location" : "Barbados",
    "joined_at" : "Nov 27, 2017",
    "profile_image" : "https://res.cloudinary.com/practicaldev/image/fetch/s--czVpA4Fy--/c_fill,f_auto,fl_progressive,h_320,q_auto,w_320/https://thepracticaldev.s3.amazonaws.com/uploads/user/profile_image/46999/256a85b6-8a6d-44c2-b7c6-f7cee7aca32f.png"
}

[Boris Jamot ✊ /]

I'll have a look at your query this evening.