https://zachpovey.dev/Recent content on Zach's BlogHugoen-ukzach@zachpovey.dev (Zach Povey)zach@zachpovey.dev (Zach Povey)Feel free to share, use, or adapt any content from this blog - a shout out would be nice!Thu, 26 Sep 2024 23:41:33 +0200https://zachpovey.dev/news/blog_launch/Thu, 26 Sep 2024 23:41:33 +0200zach@zachpovey.dev (Zach Povey)https://zachpovey.dev/news/blog_launch/<p>Hello! I&rsquo;ve just started this new blog, I hope you like it. I&rsquo;ll link a reddit thread on each article where you contact me regarding the specifics of the posts but for anything else you can contact me on <a href="mailto:zach@zachpovey.dev">zach@zachpovey.dev</a>.</p>https://zachpovey.dev/blog/the_pedestrian_yet_subtle_art_of_csv_ingestion/Mon, 23 Sep 2024 20:35:41 +0200zach@zachpovey.dev (Zach Povey)https://zachpovey.dev/blog/the_pedestrian_yet_subtle_art_of_csv_ingestion/<h1 id="the-pedestrian-yet-subtle-art-of-csv-ingestion">The Pedestrian Yet Subtle Art of CSV Ingestion</h1> <p><strong>TL;DR:</strong></p> <ul> <li>CSVs (or worse yet, excel files) offer a poor choice of data format for an API.</li> <li>Despite that, there is sometimes no escaping them.</li> <li>When this situation arises there are some <a href="#but-if-you-have-to">key principles</a> to be followed to avoid the solution becoming a constant source of busy work.</li> <li>I believe there is a <a href="#appendix-tooling">gap in the available tooling</a> for this specific use case.</li> </ul> <h2 id="background">Background</h2> <p>For personal reasons (moving country and a baby arriving) I have changed employer twice in short succession recently. Moving through companies rapidly like this allows you to see certain recurring problems more clearly sometimes. One thing I noticed is that in all three of my recent employers there was at least one process which ingested CSV or excel files that were provided by a third party via FTP or email.</p>