Sixty-six percent of data scientists describe themselves as self-taught. A little more than half said they used online courses to learn new disciplines. This relatively new discipline of data science is a terrific model of how people can adapt and take advantage of changes in employment opportunity and the importance of lifelong learning. Stack Overflow, in a survey of 64,000 developers, found 14.3% of machine learning specialists looking for a new job and 13.2% of data scientists. Why then are data scientists frustrated and spending 1-2 hours a week looking for a new job?
Jonny Brooks in a KDNuggets blog outlined four reasons that leave data scientists frustrated.
Expectations Don’t Match Reality
Analytical candidates who aren’t afraid of programming get into data science because they get excited about solving complex problems with the latest and greatest applications that have a significant impact on their business. As Johnny says, “This was a chance to feel like the work we were doing was more important than anything we’ve done before.” The reality isn’t living up to the dream because companies aren’t willing to address the complicated steps needed to make the data scientist’s work pay off.
Hiring a senior, hands-on data scientist is a good first step, but companies must be prepared to add a data engineer and a data analyst to make that data scientist productive. Data Engineers build the data pipeline and infrastructure from the ground up and the data analyst is needed to do the reporting and data requests that can bury even the best data scientists. Only then does the data scientists have the data resources and the time to do the more sophisticated analysis.
Politics Reign Supreme
The commonly assumed complaint is that the data scientist spends 80% of their time getting, cleaning and managing the data leaving only a small amount of time for doing analysis. The more accurate complaint is the successful data scientist has to deal with the interpersonal dynamics of building a data infrastructure and algorithms that highlight flaws in current processes and products. Communication and good interpersonal skills are critical to getting the data science function out of the back room without insulting someone. There will be resistance to data collection and sharing, rejection of your forecasts and insights, treating you like a simple Excel expert and giving you every problem even if it has nothing to do with the role you supposed to play. You must find friendly managers and work with them, produce value as soon as possible, communicate everything you do to as many people as possible at every level, be nice always and explain everything clearly and help everyone. You will be overwhelmed and proving you have value in the short term means you can start setting policies that will make you more productive in the long term.
You Are The Go To Person About Anything Data
Of course you know Spark, Hadoop, Hive, Pig, SQL, MySQL, Python, R, Scala, Tensorflow, A/B Testing, Natural Language Processing and Machine Learning. Because you know all of this and you have access to all of the data, you can, of course, answer any data related question. Jonny says, “If you see a job specification with all of these written on it, stay well clear.” Educating your associates and management on what you can and can’t do will be critical for your success. Matching your skills to what the company really needs may be something you can define better than the hiring manager.
Working On An Isolated Team
We see many firms building Centers of Excellence for their data science function. This can leave data scientists frustrated because they don’t interact actively enough with the line managers to determine what the best, most bottom-line oriented projects are. Yes, they can sit in an ivory tower and do interesting work but is it the most effective work for the organization? Getting into the trenches on a daily basis is where the data scientist will have the most impact.
Are you one of the data scientists frustrated by your current role? Contact the Smith Hanley Associates’ Data Science and Analytics Recruiters, Paul Chatlos at pchatlos@smithhanley.com and Nancy Darian at ndarian@smithhanley.com.