I think the the second part was kinda obvious? The moment I read this:
> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.
it was pretty obvious I was not anything like him. My intuitive answers are pretty different.
- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?
- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).
For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?
- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.
- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.
Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".
Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are:
1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be as simple as "it was the 20th request to arrive during a period of high demand, and backends couldn't keep up". The whole point of queues is so you can gracefully handle this case without overprovisioning your backends by a factor of 20x, but if the user is still going to consider this a miss anyway, you have to overprovision the backends anyway. There isn't really another way to handle this other than having spare backend capacity. Also note that in many cases the user hitting "refresh" doesn't cancel the existing queued job, it just adds another one to the queue. Which brings us to...
2.) They can turn simple failures into cascading failures. There were several postmortems that went something like "Service X became overloaded because of an unexpected flood of requests, leading to several individual replicas shutting down. This led to more requests being routed to the remaining replicas, which overloaded them too and led to all of Service X going down. When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure. SRE had to limit requests upstream and manually drain all job queues and bring Service X back cluster by cluster to restore service health."
The root principle here is that any distributed system needs a concept of backpressure. When critical downstream dependencies are overloaded, they need to pass this information back up the stack to the entry point, which needs to start denying requests from the user or do a simpler fallback that doesn't put load on the overloaded service. Naive queueing does not work, because the requests are still sitting there in the queue waiting to overload the downstream service once it becomes available again.
You can bolt backpressure onto a job queue system (by eg. rate-limiting requests to a service that has just come back up, or rate-limiting based on response time, and/or falling back to simpler algorithms), but at that point, it's a backpressure system, not a job queue. The semantics are very different from a system that guarantees eventual delivery, just not sure when. You need to be able to handle partial failures and adopt a single algorithm at multiple points within the system.
I haven't modeled it, but I wonder how far you'd get on randomizing the policy choice for concurrency limit 1. Maybe weighted by past results, but bounded to allow it to shift instead of falling permanently into a basin.
> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.
it was pretty obvious I was not anything like him. My intuitive answers are pretty different.
- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?
- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).
For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?
- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.
- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specific operation. For example, if there job starts by fetching data, and this fetch can be super slow, use 'Prefer Old' and put a timeout on the fetch part. This way your job won't be interrupted if the fetch just finally succeed minutes before next scheduled interval hit.
Oh, and re "If the Prefer Old semantics are not offered, you can’t really emulate them using the two primitives of regular scheduling and limiting concurrency." - you totally can. Set concurrency to 2, and add as a first thing: "fetch the list of jobs running; if there is anyone except me, exit right away".
Really, not that tricky at all.
1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be as simple as "it was the 20th request to arrive during a period of high demand, and backends couldn't keep up". The whole point of queues is so you can gracefully handle this case without overprovisioning your backends by a factor of 20x, but if the user is still going to consider this a miss anyway, you have to overprovision the backends anyway. There isn't really another way to handle this other than having spare backend capacity. Also note that in many cases the user hitting "refresh" doesn't cancel the existing queued job, it just adds another one to the queue. Which brings us to...
2.) They can turn simple failures into cascading failures. There were several postmortems that went something like "Service X became overloaded because of an unexpected flood of requests, leading to several individual replicas shutting down. This led to more requests being routed to the remaining replicas, which overloaded them too and led to all of Service X going down. When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure. SRE had to limit requests upstream and manually drain all job queues and bring Service X back cluster by cluster to restore service health."
The root principle here is that any distributed system needs a concept of backpressure. When critical downstream dependencies are overloaded, they need to pass this information back up the stack to the entry point, which needs to start denying requests from the user or do a simpler fallback that doesn't put load on the overloaded service. Naive queueing does not work, because the requests are still sitting there in the queue waiting to overload the downstream service once it becomes available again.
You can bolt backpressure onto a job queue system (by eg. rate-limiting requests to a service that has just come back up, or rate-limiting based on response time, and/or falling back to simpler algorithms), but at that point, it's a backpressure system, not a job queue. The semantics are very different from a system that guarantees eventual delivery, just not sure when. You need to be able to handle partial failures and adopt a single algorithm at multiple points within the system.