Questions tagged [task-queue]
With the Task Queue API, applications can perform work outside of a user request, initiated by a user request.
765
questions
132
votes
6
answers
104k
views
Background processes in Node.js
What is a good aproach to handle background processes in a NodeJS application?
Scenario: After a user posts something to an app I want to crunch the data, request additional data from external ...
- 16.3k
64
votes
3
answers
36k
views
Is there a performance difference between pooling connections or channels in rabbitmq?
I'm a newbie with Rabbitmq(and programming) so sorry in advance if this is obvious. I am creating a pool to share between threads that are working on a queue but I'm not sure if I should use ...
- 25.6k
51
votes
9
answers
70k
views
Parallel processing in PHP - How do you do it?
I am currently trying to implement a job queue in php. The queue will then be processed as a batch job and should be able to process some jobs in parallel.
I already did some research and found ...
- 4,203
36
votes
1
answer
5k
views
What is TombstonedTaskError from App Engine's Task Queue?
What does the TombstonedTaskError mean? It is being raised while trying to add a task to the queue, from a cron-job:
Traceback (most recent call last):
File "/base/python_lib/versions/1/google/...
- 168k
27
votes
2
answers
16k
views
RabbitMQ: What Does Celery Offer That Pika Doesn't?
I've been working on getting some distributed tasks working via RabbitMQ.
I spent some time trying to get Celery to do what I wanted and couldn't make it work.
Then I tried using Pika and things ...
- 2,720
24
votes
1
answer
8k
views
What is meant by 'bucket-size' of queue in the google app engine?
Google app engine task queues have configuration as (example)
<queue>
<name>mail-queue</name>
<rate>5/m</rate>
<bucket-size>10</bucket-size>
&...
- 10.2k
24
votes
1
answer
9k
views
Push vs Pull Queues on Google App Engine
So far, no one (not even the GAE docs) has been able to give me a really clear description of what the difference is between a push queue and a pull queue.
My understanding is that the Task Queue API ...
- 56.3k
23
votes
2
answers
36k
views
how to use q.js promises to work with multiple asynchronous operations
Note: This question is also cross-posted in Q.js mailing list over here.
i had a situation with multiple asynchronous operations and the answer I accepted pointed out that using Promises using a ...
- 10.5k
19
votes
3
answers
34k
views
Return data from Laravel Jobs
TL;DR: How can I return data from a queued Job without saving it anywhere and handling the situation when the job might run more than once because of retries? Or is there another way if Jobs are not ...
- 781
19
votes
1
answer
19k
views
Dynamic queue creation with RabbitMQ
I've been learning RabbitMQ various topologies, however, I couldn't find any reference to dynamic queue creation (aka Declare Queue) emitted from a producer.
The idea would be to create queues ...
- 1,315
18
votes
4
answers
5k
views
GAE: unit testing taskqueue with testbed
I'm using testbed to unit test my google app engine app, and my app uses a taskqueue.
When I submit a task to a taskqueue during a unit test, it appears that the task is in the queue, but the task ...
- 16.3k
18
votes
1
answer
2k
views
Running out of PermGen space when using DeferredTasks in GAE 1.7.3
I switched to Google App Engine Java SDK 1.7.3 recently. Since then, I am running out of PermGen space every time I am submitting DeferredTasks into the task queue.
This does not happen when the app ...
- 1,562
17
votes
5
answers
6k
views
Throttle JavaScript function calls, but with queuing (don't discard calls)
How can a function rate-limit its calls? The calls should not be discarded if too frequent, but rather be queued up and spaced out in time, X milliseconds apart. I've looked at throttle and debounce, ...
- 148k
17
votes
1
answer
7k
views
In celery, what would be the purpose of having multiple workers process the same queue?
In the documentation for celeryd-multi, we find this example:
# Advanced example starting 10 workers in the background:
# * Three of the workers processes the images and video queue
# * Two of ...
- 17.4k
16
votes
4
answers
10k
views
Where do I register an rq-scheduler job in a Django app?
I'd like to use django_rq and rq-scheduler for offline tasks, but I'm unsure of where to call rq-scheduler's ability to schedule repeating tasks. Right now, I've added my scheduling to a tasks.py ...
- 17.5k
15
votes
3
answers
21k
views
How to create multiple workers in Python-RQ?
We have recently forced to replace celery with RQ as it is simpler and celery was giving us too many problems. Now, we are not able to find a way to create multiple queues dynamically because we need ...
- 1,480
15
votes
2
answers
2k
views
How to implement a light pub-sub service on App Engine?
During his Google I/O 2009 "Offline processing on App Engine: A look ahead" presentation (video, slides), Brett Slatkin presents the Task Queue service.
He states that
Pub-sub systems maximize ...
- 66.1k
14
votes
1
answer
4k
views
with node.js on app engine, is it better to use task queues or pub/sub
We have been moving our apis from python to node. We have used task queues with our Python app engine apis. With node.js now supported on app engine, do you suggest we use task queues or cloud pub/sub ...
- 609
13
votes
3
answers
10k
views
How can I queue a task to Celery from C#?
As I understand message brokers like RabbitMQ facilitates different applications written in different language/platform to communicate with each other. So since celery can use RabbitMQ as message ...
- 155
13
votes
1
answer
9k
views
Flask with Celery - Application context not available
I have a Flask application, which is registered as follows:
APP = Flask(__name__)
APP.config.from_object('config')
I have defined a view for a URL, in which a function is called, which interacts ...
- 4,862
13
votes
1
answer
3k
views
Unsatisfactory job push performance with Python RQ
Trying to use python-rq to support the back end to our Web application, but pushing new jobs takes very long - up to 12 seconds.
The performance hit happens when executing the enqueue_call function ...
- 12.3k
12
votes
3
answers
22k
views
How to run a Laravel Job at specify Queue
I have a Job to perform send SMS to user. I want to run this job on the specify queue-name. For example, this job added to "SMS" queue. So I found a way to do this but it's exists some ...
- 888
11
votes
1
answer
3k
views
Is there a compelling reason to use an AMQP based server over something like beanstalkd or redis?
I'm writing a piece to a project that's responsible for processing tasks outside of the main application facing data server, which is written in javascript using Node.js. It needs to handle tasks ...
- 948
11
votes
3
answers
3k
views
Google App Engine: task_retry_limit doesn't work?
I have a Python GAE app.
I want my tasks to stop running or just retry once if they fail. Right now, they run forever despite what my yaml file is telling them!
Here is a queue.yaml entry:
- name: ...
- 111
11
votes
7
answers
16k
views
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build XXXXXXXXXXXX status: FAILURE
I had this error for almost an hour but found a solution resulting into a big worrying dilemma. The solution let to realization that one cannot run gcloud app deploy while there is a task in the task ...
- 710
10
votes
1
answer
7k
views
Managing Celery Task Results
I am pretty new to Celery and I thought I had read somewhere that the task results only stay around for a limited time. However my backend (redis) is getting pretty bloated after running a lot of ...
- 6,655
9
votes
2
answers
15k
views
How to force COMMIT inside function so other sessions can see updated row?
In a Postgres 12 database, I have multiple queries (SELECT, UPDATE, ...) in a function that all together take about 20 minutes to complete.
I have a check at the top that does an UPDATE if status is ...
- 93
9
votes
2
answers
15k
views
accessing bull-queue to view job stats from nodejs
I need to access bull-queue to view job stats and show on the page. I'm using bull-repl to access the queue from CLI, as follows:
> bull-repl
BULL-REPL> connect marathon reddis://localhost:...
- 469
9
votes
3
answers
4k
views
Laravel 5: Multiple workers - Cannot delete job XXXXXX: NOT_FOUND
I'm creating a website using Laravel 5. I'm using queues with beanstalkd with 2 tubes. I have 10 workers monitored by daemontools. 5 workers process tube A and B. The other 5 workers only process ...
- 664
9
votes
2
answers
3k
views
A queueing system for Perl
I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs.
I've tried IPC::...
- 146k
9
votes
1
answer
5k
views
Web Application Architecture - Job/Task Queue needed?
I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time ...
- 1,653
9
votes
3
answers
7k
views
asynchronous processing with PHP - one worker per job
Consider a PHP web application whose purpose is to accept user requests to start generic asynchronous jobs, and then create a worker process/thread to run the job. The jobs are not particularly CPU or ...
- 9,152
9
votes
3
answers
4k
views
Checking status of Task Queue in Google App Engine
I'm putting several tasks into a task queue and would like to know when the specific tasks are done. I haven't found anything in the API about call backs, or checking the status of a task, so I ...
- 8,741
9
votes
1
answer
2k
views
How to limit throughput with RabbitMQ?
Where did the question:
We are using RabbitMQ as task queue. One of specific tasks - sending notices to Vkontakte social network. They api has limit to request per seconds and this limit based on ...
- 6,331
8
votes
3
answers
9k
views
Creating a execution queue by using Task.ContinueWith?
I have several actions that I want to execute in the background, but they have to be executed synchronously one after the other.
I was wondering if it's a good idea to use the Task.ContinueWith ...
- 4,071
8
votes
2
answers
9k
views
Python: Distributed task queue for different specific workers
I am looking for a python library / framework that manages task distribution (e.g. a task queue).
However, tasks will require specialized workers: Worker A can only handle tasks of type a, workers B ...
- 481
8
votes
2
answers
1k
views
Google App Engine: bucket_size parameter on Task Queues
Google documentation on Bucket_size parameter gives just a brief definition, a Wikipedia link and an example:
bucket_size
Limits the burstiness of the queue's processing,
i.e. a higher ...
- 73.1k
8
votes
2
answers
5k
views
What's the execution model of uwsgi spoolers?
I need a task queue so that some of the heavy operations can be moved out of the uwsgi context without affecting the users. Since we're already using uwsgi app server, it'd be desirable if uwsgi ...
- 530
8
votes
1
answer
3k
views
How to correctly catch and process RQ timeouts in Python?
Trying to find a good way to catch a timeout of an RQ job, so it can be requeued after the timeout.
Basically, the correct solution would provide a way (for example, an exception handler in the ...
- 12.3k
8
votes
3
answers
1k
views
Networkx as a task queue?
I have a directed acyclic graph in networkx. Each node represents a task and a nodes' predecessors are task dependencies (a given task cannot execute until its' dependencies have executed).
I'd like ...
- 6,545
7
votes
7
answers
3k
views
Is there a way to assure FIFO (first in, first out) behavior with Task Queues on GAE?
Is there a way to assure FIFO (first in, first out) behavior with Task Queues on GAE?
GAE Documentation says that FIFO is one of the factors that affect task execution order, but the same ...
- 302
7
votes
3
answers
1k
views
NDB not clearing memory during a long request
I am currently offloading a long running job to a TaskQueue to calculate connections between NDB entities in the Datastore.
Basically this queue handles several lists of entity keys that are to be ...
- 431
7
votes
1
answer
5k
views
Is it possible to use celery for synchronous tasks?
Nearly synchronous works, too; basically, I want to delegate the data access and processing behind a web app to a task queue for most jobs. What's the fastest latency that I can consider reasonable ...
- 17.8k
7
votes
2
answers
2k
views
How should I use Celery when task results are large?
What's the best way to handle tasks executed in Celery where the result is large? I'm thinking of things like table dumps and the like, where I might be returning data in the hundreds of megabytes.
I'...
- 17.8k
7
votes
1
answer
3k
views
django-celery without an amqp server (rabbitmq)
I am using celery for distributed task processing. I wanted to deploy my work on a web-host, just to show the working of my project.
So how can i get djcelery to use a database (sqlalchemy) as backend ...
- 13.5k
7
votes
3
answers
3k
views
How can tasks be prioritized when using the task queue on google app engine?
I'm trying to solve the following problem:
I have a series of "tasks" which I would like to execute
I have a fixed number of workers to execute these workers (since they call an external API using ...
- 13k
7
votes
1
answer
353
views
Concurrent load handling in Django [closed]
I am using Django for my new web project. Its basically a Quora clone. Lets say I have a question and Upvote button below it. if around 8-9k people simultaneously click on that Upvote button(of the ...
user5042031
7
votes
1
answer
1k
views
Why do processes running on Google App Engine hang?
We have a cron job that runs every hour on a backend module and creates tasks. The cron job runs queries on the Cloud SQL database, and the tasks make HTTP calls to other servers and also update the ...
- 91
7
votes
3
answers
2k
views
Where is the huey consumer configuration?
I'm trying to get running the Huey task queue for python (it's a Celery alternative) and I'm stuck with starting the consumer with main.Configuration (as it's written in the tutorial). I know the ...
- 992
6
votes
4
answers
8k
views
What is a good alternative to Kue that works with MongoDB instead of Redis?
I am building a web app with node.js and mongodb.
I need to add delayed jobs. For example, sending users an email one month after signing up.
I haven't found an existing solution for delayed jobs ...
- 911