The Modern Data Engineer’s Playbook: Learning, Leading, And Automating With Urvang Kothari

There is no doubt that data is playing an increasingly important role in the way businesses operate today. From understanding customers to improving supply chains, organisations depend on it to make decisions more efficiently. This process is often managed by data engineers, who support these efforts. The story of Urvang Kothari, a professional who has been working with organizations to use data more effectively, reflects this trend.

Kothari’s role involves more than just building pipelines to move data. He works on designing systems that help businesses access real-time insights, automate routine tasks, and apply machine learning in practical ways. His experience includes industries such as gaming, utilities, and manufacturing, and he focuses on building cloud-based systems that are efficient, dependable, and scalable. He shared that his career has progressed from a Business Engineering role to a Senior Data Engineer. He has contributed to the use of tools like Snowflake, BigQuery, Azure, Apache Airflow, and Python, which assist organizations in managing large volumes of data. Whether it's automating the retraining of machine learning models or improving how data moves across systems, his aim is to support timely decision-making using data.

Discussing his key projects, he mentioned helping a gaming company shift from traditional systems to real-time data platforms. Using Google Cloud tools, he helped set up a system that could process and analyze data from players while they were still in the game. This allowed game developers to make quicker changes and understand how people were using their products. In another project for a utility company, Kothari set up automated workflows that could identify problems in operations before they happened, helping teams fix issues faster and reduce downtime.

Alongside the technical work, the professional also focuses on supporting his team. He has trained junior engineers, helped interns learn standard practices, and contributed to improving how projects are tracked and documented. By introducing workflows through tools like Jira and Confluence, his team was able to complete projects more efficiently and coordinate across departments. In addition to this, he has also been involved in efforts to reduce the amount of manual work done by engineers. By building data systems, he assisted in decreasing manual tasks by half. He added features that could detect errors in data processes and notify the appropriate people promptly. This shortened response times and helped maintain system availability.

However, not every success came smoothly. One of the biggest challenges, according to him, was encouraging teams to adopt new technologies. Change is never easy, especially when people are used to older systems. So, this was tackled by working closely with his team—running training sessions, offering hands-on help, and showing the benefits of new approaches through real results.

Further, stating his beliefs, he added that data engineering is as much about mindset as it is about tools. For him, learning, teaching, and improving systems through automation are all important. He has created internal training materials, shared his insights through presentations, and supported a culture of ongoing learning—these are aspects he highlights.

Now, looking towards the future, it is anticipated that data engineers will enjoy a bigger role in shaping how organisations use technology. In this, automation, curiosity, and teamwork will be key. As technology keeps changing, the ability to adapt, learn, and help others do the same will matter more than ever.