@Abhi Agarwal [redacted] monorepo is storing ~$1,000/month of Git LFS objects, is that expected?
Fun message to get at 3PM on a friday.
I store seed data in a DuckDB file locally in said monorepo, enabling local development for times-series data. This file is about a gigabyte. My first thought getting this message was, there is no way in hell a GIGABYTE costs a band a month.
It doesn’t. I was then informed that retrieval of said object is billed and charged.
Ah. My checkout job has been taking 50 seconds in CircleCI. It’s driven me mad. I’ve wasted a lot more time than I should probably admit to trying sparse checkouts, blobless clones, all techniques possible to try and minimize the checkout time. I resigned to blame it on the merging and rewriting of history I did across several repos to unify them into said monorepo with git-filter-repo magic. That checkout was slower than the actual time it takes to run our jobs.
And that’s when I learned that indeed, on every single CCI pipeline, on every single job that calls git checkout; we had been downloading this DuckDB file. A file only used for local development.
Setting GIT_LFS_SKIP_SMUDGE=1 instantly solved the problems. Checkouts are now 2 seconds at the most. And presumably, next month, I won’t be getting the same message.
I wish I had some profound lesson here, a tale for what I learned from this experience, a promise that I will be more vigilant in the future. But really, the only thing I feel is a prolonged sense of, god dammit.