This coming October, Israel Tech Challenge Hi-Tech Academy <itc> will be launching its first Data Engineering Program. Based on our 6 years of experience in training high quality Data Scientists, our new program will be teaching the core technologies and skills that will upgrade OOP programmers to in-demand Data Engineers. So, we spoke with one of our Data Science graduates, Juan Amari, and asked if he could share some insights from his previous experience working as a Data Engineer. Here’s what he had to say.
What would you say is the main difference between a Data Scientist and a Data Engineer in Israel?
Data Scientist and Data Engineer are two terms for which each company has an entirely different definition. More often than not Data Scientists tend to be focused on experimentation and hypothesis validation whilst Data Engineers are more focused in building and maintaining high throughput pipelines. That’s the general by-the-book opinion though, in truth there’s a very visible overlap between tasks where Data Engineers work with Data Scientists in experimentation, and vice-versa.
What do you like about your work in Data Engineering in Israel?
I’m not just the ETL guy, I made it very evident from the beginning that I didn’t care about building ETLs but rather build high complexity machine-learning and high throughput data-processing systems. I very much enjoy the overlap between Data Science and Data Engineering and I personally thrive in that specific overlap. Data Engineering is a flexible role and you can find what you’re looking for by asking each company what they see a data engineer doing. I personally enjoy doing both data engineering and data science.
How would you say <itc> helped you succeed in this field you’re in?
<itc> helped me bring value in terms of analytical experience. I’m not just building systems with little knowledge about the end result, but rather full pipelines with complex mechanisms that I constantly contribute to in every step, including the product.
What good advice would you give an OOP programmer that is considering to upgrade their career and become a Data Engineer?
A Data Engineer still remains a very flexible role that meddles with real-time as well as batch processes. It is not very far away from a Software Engineering role by any means, but it requires putting thought into pipelines and data sources instead of micro services. Think about dependencies and how dependencies will be fulfilled; What if one is missing? How will you know it’s missing? How will you fulfill such a dependency between processes? This type of thinking really defines this field of work.
With your experience so far, what keyword do you think a Data Engineer must include in their CV?
Knowledge in batch or stream processing tools is key. So is experience in workflow orchestration and data processing. SQL is part of the toolbox of any engineer and any data scientist, but more complex data processing methods are important as well. These may be Apache Spark, Hadoop or others.
Juan Amari
Can you describe for our readers what type of work you do in your company and why it is essential for the progress being made?
At my team in Snyk we augment our product capabilities by leveraging the value of raw data. We process it and run aggregations and create statistical models which give room for previously unseen insights in the world of cloud native security. As for why it’s essential: we’re in an era where computer power has reached a point where it is possible to process quantities of data and run algorithms that were previously impossible to run. Such is the case with deep learning. Having people that understand how to store and process this data is an extremely important step in order to bring insights in the form of aggregations, reports, and to enable the capability of running complex algorithms that require big quantities of data to bring good results to the table.