Mini Viva - made even easier by the fact that their entire discography was three singles and, later, a leaked album.
As far as I know, Hospitality haven't officially broken up, but they only did two albums in 2012 and 2014 and I liked both when they came out (and still really like the second). (Although I had no idea before checking Wikipedia just now that they put out an EP in 2008, so maybe I can't count them.)
There's artists that I knew at or prior the debut album that have since broken up that I can kind of count, depending on your definition of "followed". Like Sally Shapiro (who I think count as a duo and not a single singer) - I was interested in their early releases and bought their first album, and I've circled back to their subsequent albums despite not being aware of them when they came out.
I was going to say The Streets, with the caveat that they're probably more of an artist than a band, but apparently they did a collaborative mixtape in 2020 with all new material that I completely missed. But I did hear all of their prior albums (five between 2002 and 2011) pretty much when they came out, albeit probably only once for the last one. He was also one of a small handful of people I followed on Twitter for the month or so I actively used it back in 2009ish.