Quoted from my another answer here:
Historically, RFC 2616, published at 1999, was the most-referenced HTTP 1.1 specs. Unfortunately its description on idempotency was vague, that leaves room for all these debates. But that specs has been superseded by RFC 7231. Quoted from RFC 7231, section 4.2.2 Idempotent Methods, emphasis mine:
A request method is considered "idempotent" if the intended EFFECT ON THE SERVER of multiple identical requests with that method is the same as the effect for a single such request. Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent.
So, it is written in the specs, idempotency is all about the effect on the server. The first DELETE returning a 204 and then subsequent DELETE returning 404, such different status code does NOT make the DELETE non-idempotent. Using this argument to justify a subsequent 204 return, is simply irrelevant.
OK so it is not about idempotency. But then a follow-up question may be, what if we still choose to use 204 in subsequent DELETE? Is it OK?
Good question. The motivation is understandable: to allow the client to still reach its intended outcome, without worrying about error handling. I would say, returning 204 in subsequent DELETE, is a largely harmless server-side "white lie", which the client-side won't immediately tell a difference. That's why there are people doing that in the wild and it still works. Just keep in mind that, such lie can be considered semantically weird, because "GET /non-exist" returns 404 but "DELETE /non-exist" gives 204, at that point the client would figure out your service does not fully comply with section 6.5.4 404 Not Found.
But then, the intended way hinted by RFC 7231, i.e. returning 404 on subsequent DELETE, shouldn't be an issue in the first place. Many more developers chose to do that. That is presumably because, any client which implements HTTP DELETE (or any HTTP method, for that matter), would not blindly assume the result would always be successful 2xx. And then, once the developer starts to consider the error handling, 404 Not Found would be one of the first errors that comes into mind. At that point, he/she would hopefully draw a conclusion that, it is semantically safe for an HTTP DELETE operation to ignore a 404 error. Problem solved.