I had big plans for this week's newsletter. I had a list of topics that I'd gathered as the week went by and I was looking forward to my day off on Friday when I would finally have time to write them up. I was going to make myself a cup of milky black tea, get comfortable on the couch and write. Blissful. If things had gone that way, you'd have received this newsletter two days ago.
So what happened? The Udacity data pipelines project is what happened. Some context as most of you are probably confused: I've been taking an online course in data engineering on udacity.com since the end of January. Up until Friday things were going well. I was three weeks ahead of schedule and I was finding the content interesting but not too challenging. Come Friday, I was pretty confident that I could get a piece of course work done and have time to write. Unfortunately, Udacity had different plans for me.
Without getting into too much detail, the goal of the project was to build a data pipeline using Apache airflow. The project instructions were straightforward enough but it wasn't long before I got stuck. And stuck in the most awful way, especially when it comes to programming: things weren't working and the error message was useless (nor were there any logs, for the programmers reading this). I turned to the internet but couldn't find much documentation. Perhaps airflow is not as popular as Udacity led me to believe?
After avoiding the Udacity help pages because I wanted to work out the solution for myself, I eventually gave in and found that other students had encountered the same problem. Luckily someone had posted a solution*, but it was underwhelming. Something along the lines of: "sometimes it works, sometimes it doesn't, try this if it doesn't". I was ready to scream. And perhaps I should have because what followed was not pretty.
By 6pm, I was in a foul foul mood. I hadn't been outside all day. I'd spent hours on the project and it wasn't done. Worse, I knew I still had a few errors to work out, and who knew how long that would take. Obviously, I hadn't done any writing. The kitchen was a mess. Suddenly, everything was crap. It was as if this Udacity assignment was bringing out all the frustration and stress of these weeks of lockdown. Or perhaps, because of the lockdown, the stress of this assignment had suddenly taken on gigantic proportions. I'm not sure.
I wish I could say that I went for a walk and felt a lot better. I did go for a walk (Tom's suggestion) and did feel a tiny bit better, but the cloud only really lifted this morning. After which, I went on a beautifully long and cleansing bike ride. I still haven't submitted the assignment, but I'm not going to stress about it - I'm still more than two weeks ahead of schedule anyway.
Wishing you all good physical and mental health in these stressful times 💜
*The error was due to an airflow plugin not being found...despite it having worked at the start of the project 🤷♀️