Like last time in 2018, we do the Top Programming Languages (TPL) to code again, this time, in 2019! After reviewing the past TPL posts, we simplify the TPL criteria into 2 categories:
- Popularity – how popular the language is
- Career Value – how does the language help you develop your career
Language Popularity
Popularity as our ranking criteria is based on “a higher popularity means a bigger community”. So we can find resources like tutorials, code samples, tips and guides easier. We use PopularitY of Programming Language Index (PYPL) as our reference to create following table:
| Rank (2018 Rank) | Language | Share (2018 Share) | Change |
|---|---|---|---|
| 1 (2) | Python | 26.0% (19.3%) | +6.7% |
| 2 (1) | Java | 21.4% (21.2%) | +0.2% |
| 3 (4) | Javascript | 8.3% (7.9%) | +0.4% |
| 4 (5) | C# | 7.6% (7.5%) | +0.1% |
| 5 (3) | PHP | 7.4% (8.0%) | -0.6% |
And we can visualize the top 5 languages using Plot.ly, likes the way we analyzed sales data in Google Store revenue prediction.
top5_lang_df = pd.DataFrame(columns=['year', 'language', 'share'])
years = [2017, 2018, 2019]
languages = ['Java', 'Python', 'PHP', 'C#', 'JavaScript']
shares = [[22.7, 21.2, 21.4], [15.7, 19.3, 26.0], [9.3, 8.0, 7.4], [8.3, 7.5, 7.6], [7.9, 7.9, 8.3]]
lang_index = 0
year_index = 0
for year in years:
for language in languages:
share = shares[lang_index][year_index]
top5_lang_df = top5_lang_df.append({'year': year, 'language': language, 'share': share}, ignore_index=True)
lang_index += 1
year_index += 1
lang_index = 0
data = []
for language in languages:
y_values = top5_lang_df.loc[top5_lang_df['language'] == language]['share'].values
y_labels = ["{}%".format(member) for member in y_values]
trace = go.Bar(
x = top5_lang_df.loc[top5_lang_df['language'] == language]['year'].values,
y = y_values,
text=y_labels,
textposition = 'auto',
name=language
)
data.append(trace)
layout = go.Layout(
title = "Top 5 Programming Languages",
xaxis=dict(
title='Year'
),
yaxis=dict(
title="Share (%)"
)
)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig)
Google Trends in 2018
Besides the top 5 languages, we can use word cloud and Google Trends to display other languages’ popularity in 2018. But first of all, let’s install pytrends package before running following codes:
from pytrends.request import TrendReq
import matplotlib.pyplot as plt
from wordcloud import WordCloud
language_list = ["Python", "C", "Rust", "Java", "JavaScript", "C#", "PHP", "R", "Go", "Assembly",
"Objective C", "Swift", "VBA", "Visual Basic", "Perl", "Ruby", "HTML", "SQL", "CSS", "MatLab", "Kotlin", "Scala", "react", "angular", "vue js",
".NET", "node js", "Delphi", "COBOL"]
pytrends = TrendReq(hl='en-US', tz=0) #use UTC +0000 timezone
lang_dict = {}
for language in language_list:
general_search_term = "programming language"
language_search_term = "{} tutorial".format(language)
keyword_list = [language_search_term, general_search_term]
pytrends.build_payload(keyword_list, cat=0, timeframe='2018-01-01 2018-12-31', geo='', gprop='')
int_o_t = pytrends.interest_over_time()
int_o_t['p_ratio'] = int_o_t[language_search_term]/int_o_t[general_search_term]
lang_dict.update({language: int_o_t['p_ratio'].mean()})
w = WordCloud(width=800,height=600,mode='RGBA',background_color='white').fit_words(lang_dict)
plt.figure(figsize=(12,9))
plt.title("Programming Languages in Google Trends 2018")
plt.imshow(w)
plt.axis("off")
plt.show()
Here we go:
Please note that other than the top 5 languages we mentioned earlier, SQL is another hot language as well. Although we never consider SQL as a programming language. It is inevitable to use when we work on an application or a system with a database.
Now we go back to the top 5 languages in 2019. They are no much different than the top 5 in 2018. Just our favorite language, Python, got a larger change (+6.7%) then other four. And Python is the most popular language from Google Trend 2018. Then why Python is so fast-growing? We come up with following 3 reasons:
- Ease of Use – a long time reader of this blog should know how easy Python is to perform different tasks. Python reads like English and it is straight forward, which help beginners adopt the language easily.
- Coverage of Machine Learning and AI – there are a lot of Machine Learning libraries available for Python, and once again, our readers should know, we use Python to solve all the machine learning challenges here :]]
- Large community – we can take a look at the Python Package Index (PyPI, please don’t confuse the term with PYPL, it is a capital letter “I” at this end), there are 165K+ Python projects available already in a wide range of categories. No matter you are working on a game project, financial project, web project or other, you can always find related packages and discussion groups for help.
Language Career Value
Other than the language popularity, we should consider on which language can help us develop our career most. Since this kind of research result should vary according to job market location. We use the US’s job market, where many giant technology companies are running, as our research location. Then we use major job search engines (Indeed, Glassdoor and Monster) in the US to search developer jobs. Thus we can record the job counts and salaries for our finding.
We used BeautifulSoup to scrap information from Indeed last time, so we can reuse the technique again. This time, we apply it on 3 job search engines. Let’ start from Indeed:
search_keyword_arr =["Java", "Python", "HTML", "SQL", "JavaScript", "C#", "PHP", "CSS", ".NET", "angular", "react"]
location = ""
url_prefix = "https://www.indeed.com"
pre_fix_text = "Page 1 of "
indeed_developer_jobs = []
for search_keyword in search_keyword_arr:
params = {
'q':search_keyword+' developer',
'l':location
}
url = url_prefix + "/jobs?"+urllib.parse.urlencode(params)
html = urllib.request.urlopen(url)
soup = BeautifulSoup(html, 'lxml')
search_count = soup.find('div', {'id':'searchCount'})
salary_title_txt = soup.find('div', {'id':'univsrch-salary-title'}).get_text()
salary_txt = soup.find('p', {'id':'univsrch-salary-currentsalary'}).get_text()
salary_value = salary_txt[:salary_txt.find(' per year')].replace('$', '').replace(',', '')
search_count_txt = search_count.get_text().strip()
if search_count_txt.startswith(pre_fix_text):
job_count_txt = search_count_txt[len(pre_fix_text):search_count_txt.find('jobs')]
job_count_value = int(re.sub(',','', job_count_txt))
indeed_developer_job = {"search engine":"Indeed", "language":search_keyword, "job count":job_count_value, "salary": salary_value}
indeed_developer_jobs.append(indeed_developer_job)
print("Keyword:{}\tJob count:{}\t{}:{}".format(search_keyword, job_count_txt, salary_title_txt, salary_txt))
Keyword:Java Job count:27,939 Java Developer salaries in United States:$102,430 per year
Keyword:Python Job count:25,846 Python Developer salaries in United States:$122,654 per year
Keyword:HTML Job count:15,383 Web Developer salaries in United States:$76,253 per year
Keyword:SQL Job count:29,565 SQL Developer salaries in United States:$84,555 per year
Keyword:JavaScript Job count:27,197 Javascript Developer salaries in United States:$111,243 per year
Keyword:C# Job count:20,880 .NET Developer salaries in United States:$91,681 per year
Keyword:PHP Job count:9,280 PHP Developer salaries in United States:$89,089 per year
Keyword:CSS Job count:17,491 Developer salaries in United States:$97,116 per year
Keyword:.NET Job count:13,997 .NET Developer salaries in United States:$91,681 per year
Keyword:angular Job count:11,079 Front End Developer salaries in United States:$108,408 per year
Keyword:react Job count:11,415 Developer salaries in United States:$97,116 per yearNow we go for Glassdoor:
url_prefix = "https://www.glassdoor.com"
glassdoor_developer_jobs = []
for search_keyword in search_keyword_arr:
url = url_prefix + "/Job/us-{}-developer-jobs-SRCH_IL.0,2_IN1_KO3,{}.htm".format(re.sub('#','', search_keyword) , len(search_keyword+' developer')+3)
req = urllib.request.Request(url, headers={'User-Agent' : "Magic Browser"})
html = urllib.request.urlopen( req )
soup = BeautifulSoup(html, 'lxml')
search_count = soup.find('p', {'class':'jobsCount'})
search_count_txt = search_count.get_text().strip()
search_count_value = int(search_count_txt[:search_count_txt.find('Jobs')].replace(',', ''))
url = url_prefix + "/Salaries/us-{}-developer-salary-SRCH_IL.0,2_IN1_KO3,{}.htm".format(re.sub('#','', search_keyword) , len(search_keyword+' developer')+3)
req = urllib.request.Request(url, headers={'User-Agent' : "Magic Browser"})
html = urllib.request.urlopen( req )
soup = BeautifulSoup(html, 'lxml')
salary_detail = soup.find('div', {'class':'OccMedianBasePayStyle__payDetails'})
if (salary_detail == None):
salary_detail = soup.find('div', {'id': 'MeanPay_N'})
if (salary_detail == None):
salary_detail_txt = "N/A"
salary_value = 0
else:
salary_detail_txt = salary_detail.get_text().strip()
salary_value = int(salary_detail_txt.split("$",1)[1].replace(",","").replace("/yr", "").replace("*",""))
glassdoor_developer_job = {"search engine":"Glassdoor", "language":search_keyword, "job count":search_count_value, "salary": salary_value}
glassdoor_developer_jobs.append(glassdoor_developer_job)
print("Keyword:{}\tJob count:{}\t{}".format(search_keyword, search_count_txt, salary_detail_txt))
Keyword:Java Job count:62,454 Jobs Average Base Pay$88,116/yr
Keyword:Python Job count:57,045 Jobs Average Base Pay$92,000/yr
Keyword:HTML Job count:11,002 Jobs $66,912*
Keyword:SQL Job count:22,496 Jobs Average Base Pay$84,779/yr
Keyword:JavaScript Job count:23,989 Jobs Average Base Pay$72,500/yr
Keyword:C# Job count:6,877 Jobs Average Base Pay$95,052/yr
Keyword:PHP Job count:8,471 Jobs Average Base Pay$93,987/yr
Keyword:CSS Job count:9,223 Jobs N/A
Keyword:.NET Job count:4,546 Jobs Average Base Pay$95,052/yr
Keyword:angular Job count:9,751 Jobs N/A
Keyword:react Job count:7,707 Jobs N/AAnd finally, we have Monster:
url_prefix = "https://www.monster.com"
monster_developer_jobs = []
for search_keyword in search_keyword_arr:
params = {
'q':search_keyword+' developer'
}
url = url_prefix + "/jobs/search/?"+urllib.parse.urlencode(params)
html = urllib.request.urlopen(url)
soup = BeautifulSoup(html, 'lxml')
search_count = soup.find('h2', {'class':'figure'})
search_count_txt = search_count.get_text().strip()
search_count_txt = (re.search( "[^\(](.+?)(?=Jobs)", search_count_txt)).group()
monster_developer_job = {"search engine":"Monster", "language":search_keyword, "job count":int(search_count_txt), "salary": 0}
monster_developer_jobs.append(monster_developer_job)
print("Keyword:{}\tJob count:{}".format(search_keyword, search_count_txt))
Keyword:Java Job count:82576
Keyword:Python Job count:40367
Keyword:HTML Job count:19999
Keyword:SQL Job count:37206
Keyword:JavaScript Job count:32062
Keyword:C# Job count:23025
Keyword:PHP Job count:3353
Keyword:CSS Job count:21743
Keyword:.NET Job count:16587
Keyword:angular Job count:11362
Keyword:react Job count:10006 Please note that there is no salary information from Monster.
Language EDA on Job Market
Now we have data from 3 job search engines, we can use it to plot a Job Market chart. First, let’s transform the data into a data frame.
developer_jobs = indeed_developer_jobs+glassdoor_developer_jobs+monster_developer_jobs
developer_jobs_df = pd.DataFrame(developer_jobs)
Then we calculate the mean of job count and salary for different languages.
language_jobcount_df = (developer_jobs_df.groupby(["language"])["job count"].mean().reset_index()).sort_values("job count", ascending=False
developer_jobs_wo_zero_df = developer_jobs_df.loc[developer_jobs_df['salary'] !=0]
language_salary_df = developer_jobs_wo_zero_df.groupby(["language"])["salary"].mean().reset_index()
language_jobcount_salary_df = language_jobcount_df.merge(language_salary_df, on=["language"], how='left')
And we plot a chart using plotly.
trace = go.Bar(
x = language_jobcount_salary_df["language"],
y = language_jobcount_salary_df["job count"],
name="Job Count",
marker=dict(color = "royalblue")
)
trace2 = go.Scatter(
x = language_jobcount_salary_df["language"],
y = language_jobcount_salary_df["salary"],
yaxis='y2',
name="Salary",
marker=dict(color = "orangered")
)
layout = go.Layout(
title = "Programming Lanaguage on Job Market",
xaxis=dict(
title="Language",
tickangle=45,
),
yaxis=dict(
title="Job Count"
),
yaxis2=dict(
title="Salary per year",
titlefont=dict(
color="orangered"
),
tickfont=dict(
color="orangered"
),
overlaying='y',
side='right'
),
legend=dict(orientation="h"),
width = 800,
height = 500
)
data = [trace, trace2]
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, config={'showLink': True})
Here it comes:
Conclusion for TPL 2019
Well, we can see Java is still the most wanted job on the job market. But Python is the runner-up in both job count and salary per year. Furthermore, Python is the most popular language in term of Google Trend and PYPI. So we declare Python as our language to code in 2019. That is why we keep prompting people to explore in Python. (And I have been saying this kind of thing since 2017 :]] ) Besides our beloved language, Python, you may notice the Angular developer is the highest paid job among others. Frontend languages, React and CSS, also have higher salary. We may awake the rise of frontend languages nowadays. Those frontend languages can now handle many operations that only backend modules handled in the past.
For career development in 2019, I would suggest to sharpen up and specialize in either Python or Frontend languages. Being a full stack developer is fun, like being a lovable all-rounded bard in a fantasy role-playing game. But there is always the great fighter or mage killing a dragon to save the day. So be focus on one skillset, in order to make good your development career in 2019!
What have we learnt in this post?
- Python is our language to code in 2019
- Frontend languages are more powerful and demanding nowadays
- We use what we have learnt in past (Word Cloud, Plotly and BeautifulSoup) to analyze our findings
Pingback: React Frontend and Flask Backend for Weather Forecast – Part 2