Originally posted here.
Data Science is often advertised as one of the best jobs in the world. Many companies are talking about AI. There are hundreds of different MOOCs for thousands of people who want to work as data scientists. Recruiters poach people, enticing them with better salaries and benefits. New SOTA approaches regularly appear and break records. You can hear about cool applications like Alpha Zero or GPT-3 from many sources. People constantly use virtual assistants and other technical wonders.
But… is everything really so great for those who really work in the field? Well, not exactly.
In fact, many data scientists change jobs frequently. Data scientists face many problems, and it is often easier to get a new job than fix flawed processes in a company. As a result, it is common to work in a company for 1–3 years and then look for a better future. Though it is common for IT jobs in general.
So, does this mean we are cursed to repeat the same cycle again and again? Personally, I don’t think so. I believe that the data science field is great despite all the problems and want to blogpost to inspire hope in you.
Let’s start by looking at the irritating problems which people face. And we will start with actually finding a job.
If you are a new person in the field and try to get a job, you’ll find it quite difficult. Thanks to lots of MOOCs and, in general, a high interest in the field, there is a huge supply of people who want to become data scientists. In response, companies raise the plank higher and higher. You need to know statistics and math, machine learning theory and practice, programming and algorithms, participate in competitions and hackathons, have pet projects, contribute to open-source, publish papers, and have some relevant job experience to pass an interview.
By the way, it took me ~8 months of studying and a lot of failed interviews to get the first job as a data scientist.
If you are an experienced person looking for a job, then it may be even more difficult because the range of topics that could be raised during an interview becomes even broader, and at the same time, in-depth knowledge of specific areas may be required.
So, thanks to skills, experience and a bit of luck, you got a new job. You start your first working day with great expectations… which often face a harsh reality.
Chances are you won’t be doing any ML at all at the beginning. You’ll likely start with such “exciting” activities as writing SQL queries for ad hoc analysis. It is possible you’ll do a lot of routine manual tasks, which aren’t automated for some reason. You’ll find out that only in courses you have a lot of high-quality data and can happily train models. It turns out that sometimes there is no or little data, and even if it is present, it may have low quality.
Well, at least you’ll learn a lot of cool things? Yes, sure. You’ll learn a lot of new things. By yourself. Sometimes you are the only DS in the company, sometimes there are other people, but they won’t or can’t teach you.
Okay, but there are still machine learning projects there? Yes, there are; you’ll be able to train models just fine.
It is just that training machine learning models sometimes is ~10% of the project. And often you’ll use the most simple models.
Also, the business won’t tell you “train a model with 90% accuracy”, the task likely sounds “we want to increase the sales of this product, do it”. It is also quite possible that your managers don’t really understand the capabilities and the limitations of machine learning and it is up to you to prove your point. It is up to you to explain that you can’t make a forecast for 5 years based on the data from the last two months.
And even if you did everything right, it is possible that the project on which you have worked for months, in which you have invested your soul and wished to see the results, will be closed because the higher-uppers decided it.
But even if the job is fine, you have great colleagues, good managers, successful projects, and learn a lot of new things, not everything is fine. Data Scientists are sometimes considered to be bad guys.
Who used personal data for advertising? Data Scientists.
Who develops surveillance and face recognition? Data Scientists.
Who is accused of making biased or not representative models? Data Scientists.
Sooooooo… if things are that bad, why do we continue working like this? First of all, I may have exaggerated a bit. Or a lot. Usually, all aforementioned things don’t happen at one company. And some points aren’t even negative for some people.
But what is more important, there are a lot of upsides and positive things.
This Reddit post really struck a chord with me. Indeed, those who work in IT (DS is a part of it) often have higher salaries than people in most other professions (of course, there are always those who earn much more, but here I talk about the average situation).
Additionally, I want to quote the book “Rise: 3 Practical Steps for Advancing Your Career, Standing Out as a Leader, and Liking Your Life.”
Once when flying across some vast expanse watching an especially beautiful sunset, the owner of the private plane stood behind the pilots, and said: “Is this what I pay you for, to sit back and relax in this $50 million jet watching the sun set?” to which the grizzled captain said, “No, you pay us to deal with all your bags, change your hotels, fix screwed-up rental car reservations, deal with catering mix-ups, get the airplane fixed, vacuum all the dog hairs up, and crap like that. This part we do for free.”
Once you realize that your job is both your job description and dealing with all the crap that gets in the way of your doing your job description, and that what you are actually getting paid for is dealing with the crap, not the enjoyable parts, it all makes more sense.
We really have a high degree of freedom. DS don’t just change jobs due to uncomfortable conditions but also because they want to pursue their interests and dreams. The field of ML and AI provides endless possibilities. You only need to improve your skills.
DS is used in most industries: banks, retail, telecom, medicine, and many others. We can stick to a single industry for the whole career and become experienced domain experts, or we can move between industries when we are bored.
Sick of training gradient boosting? You can try geo-data, time-series, graph approaches, recommender systems.
Build chat-bots or language translation systems in NLP.
Detect missing people, classify birds, find defects, generate cute images in CV.
Work on speech-to-text models or improve speech synthesis in audio.
Don’t want to train models at all anymore? Try to be a statistician, visualization expert, business expert, data engineer, ml researcher, or choose any other specialization.
Want to have more impact? Become tech-lead, product owner, project manager and orchestrate the development of multiple products at once.
You like your job but feel you don’t get enough experience? Contribute to open-source projects, create pet-projects, complete courses, participate in conferences, and do volunteer projects.
You can teach other people; you can bring revenue for the business, you can do research, you can build unique products with weak AI.
You can work in startups or big companies, academies, non-profit, social volunteer projects, you can be a consultant or a freelancer.
Got bored working in your country? Find a remote position or work in another country.
And of course: you want a job to be simply a job and have a life outside of it? There are enough possibilities to find a nice, stressless job and comfortably work there.
Do the advantages outweigh the disadvantages? I can’t answer for other people — but for me, working as a Data Scientist is definitely worth it. I have worked in ERP-consulting on junior and middle positions— trust me, it was much worse, and the pay was much lower.
There is a limited number of professions with a good salary in many countries, and Data science is one of them.
The most positive things for me are work mobility, good communities, freedom in choosing the job, and the salary. Maybe in the future, I’ll change my career or specialization or switch to manager roles, but it is okay for now.