Questions tagged [worker]
In software, the term worker usually describes a long-running process being executed in a separate thread from an application's main thread of execution. Consider [tag:web-worker] for javascript.
1,275
questions
192
votes
32
answers
563k
views
nginx.service failed because the control process exited
nginx.service failed because the control process exited
$ systemctl status nginx.service
nginx.service - Startup script for nginx service
Loaded: loaded (/usr/lib/systemd/system/nginx.service; ...
- 1,921
47
votes
4
answers
17k
views
What are the consequences of disabling gossip, mingle and heartbeat for celery workers?
What are the implications of disabling gossip, mingle, and heartbeat on my celery workers?
In order to reduce the number of messages sent to CloudAMQP to stay within the free plan, I decided to ...
- 6,883
44
votes
1
answer
25k
views
multiple worker/web processes on a single heroku app
Is there some way to configure multiple worker and/or web processes to run in the single Heroku app container? Or does this have to be broken up into multiple Heroku apps?
For example:
worker: node ...
- 835
28
votes
1
answer
12k
views
Starting multiple upstart instances automatically
We use PHP gearman workers to run various tasks in parallel. Everything works just fine, and I have silly little shell script to spin them up when I want them. Being a programmer (and therefore lazy)...
- 941
24
votes
4
answers
20k
views
What is the appropriate number of Gunicorn workers for each Amazon Instance Type?
I'm currently trying to figure out what the appropriate number of workers is for each Amazon Instance Type. I used to run one Gunicorn worker, however that proved to be quite slow.
Many developers ...
- 7,208
23
votes
4
answers
70k
views
Cannot find module 'worker_threads' error in production
I created a node js project that utilizes worker threads. The code works fine when I run npm start in VS Code. But when build and copied in Ubuntu server, it show "Cannot find module 'worker_threads'"....
- 231
23
votes
1
answer
11k
views
Gunicorn with multiple workers: Is there an easy way to execute certain code only once?
If I start gunicorn with -w 4 with an app with the content
print 'hello'
it will print four times 'hello'. Is there some way to coordinate this in such a way that only one 'hello' is printed? I want ...
- 5,861
22
votes
5
answers
8k
views
Resolving imports using webpack's worker-loader in Jest tests
I'm writing a Jest test and am having trouble resolving this import which uses webpack's worker-loader
import ImageInfoWorker from 'worker-loader?name=image_info!@WORKERS/image-info';
I have some ...
- 4,308
20
votes
2
answers
27k
views
NestJS run worker in a separate process
I am implementing NestJS worker, queues, using Bull.
According to the documentation, both the worker and the server (will) run in a same "process", but I want to run the worker in a separate ...
- 3,655
17
votes
1
answer
40k
views
Amazon SQS how to control the number of retries
I am configuring a beanstalk worker environment to deal with periodic tasks
When the requested URL is not responding with status 200 Elastic Beanstalk will put the task again in the queue.
How can I ...
- 37.6k
17
votes
2
answers
13k
views
Service Worker : How to handle a 302 redirect response
I installed a service worker on my application, it gets installed well, activated well, and the caching is ok too.
But when the caching is done when I click on a page that is a 302, it tells me:
...
- 170
16
votes
1
answer
13k
views
How to use AddHttpClient in .NET 6 Worker Service
Is there an alternative way to add the "AddHttpClient()" setting in the .NET 6 Worker Service template?
In Minimal APIs for example we have available:
var builder = WebApplication....
- 341
16
votes
2
answers
21k
views
How to configure Apache Spark random worker ports for tight firewalls?
I am using Apache Spark to run machine learning algorithms and other big data tasks. Previously, I was using spark cluster standalone mode running spark master and worker on the same machine. Now, I ...
- 161
16
votes
4
answers
15k
views
How to close Smtp connection in SwiftMailer
I use SwiftMailer to send emails from a gearman worker process. I'm using the Swift_SmtpTransport class to send emails.
The problem is that if this worker process stays idle for sometime, the ...
- 1,541
16
votes
2
answers
11k
views
How does Celery work?
I have recently started working on distributed computing for increasing the computation speed. I opted for Celery. However, I am not very familiar with some terms. So, I have several related questions....
- 353
16
votes
4
answers
7k
views
Worker pools and multi-tenant queues with RabbitMQ
I work on a web application that is a multi-tenant cloud based application (lots of clients, each with their own separate "environment", but all on shared sets of hardware) and we're introducing the ...
- 2,292
15
votes
2
answers
7k
views
multiple methods per sidekiq worker
I don't get it.
Per Sidekiq documentation, each worker (mine is called FeedWorker) can only contain one method called perform. Well, what if I want to run mulitple methods through the same worker?
...
- 1,584
14
votes
3
answers
5k
views
Java Performance Processes vs Threads
I am implementing a worker pool in Java.
This is essentially a whole load of objects which will pick up chunks of data, process the data and then store the result. Because of IO latency there will ...
- 1,928
14
votes
3
answers
21k
views
Spark CollectAsMap
I would like to know how collectAsMap works in Spark. More specifically I would like to know where the aggregation of the data of all partitions will take place? The aggregation either takes place in ...
- 153
14
votes
3
answers
8k
views
JavaFX SwingWorker Equivalent?
Is there a JavaFX equivalent to the Java SwingWorker class?
I am aware of the JavaFX Task but with that you can only publish String messages or a progress. I just want to call a method in the GUI ...
- 815
13
votes
2
answers
3k
views
Running PHP workers on AppFog
I searched and searched and searched but didn't find any resources on that.
Is there any way to run a worker in a PHP app on AppFog?
I've only found instructions for running Ruby, Node.js and Python ...
- 835
13
votes
2
answers
16k
views
Get worker id from Gunicorn worker itself
I run multiple gunicorn workers with workers=4 setting. So as I understand I have 5 different processes: one gunicorn master process and 4 other worker processes. Can I get information about which ...
- 6,744
13
votes
2
answers
15k
views
How to setup event log for .NET Core 3.0 Worker Service
I'm working with the new Worker Service app template with .NET Core 3.0 Preview and am trying to add event logging using the AddEventLog method. However, I cannot see any of my logs via the Event ...
- 778
13
votes
2
answers
8k
views
Python What is the difference between a Pool of worker processes and just running multiple Processes?
I am not sure when to use pool of workers vs multiple processes.
processes = []
for m in range(1,5):
p = Process(target=some_function)
p.start()
processes.append(p)
for p in ...
- 1,634
13
votes
3
answers
3k
views
How to run Node Cluster on windows?
Anyone know how to run Node Cluster on windows? I haven't been able to find any articles on the web and cannot seem to solve this problem:
events.js:160
throw er; // Unhandled 'error' event
...
- 8,917
12
votes
3
answers
29k
views
worker queue for nodejs?
I am in the process of beginning to write a worker queue for node using node's cluster API and mongoose.
I noticed that a lot of libs exist that already do this but using redis and forking. Is there ...
- 57.5k
12
votes
1
answer
14k
views
Worker, Threads & Pool size using Puma
If I have a server with 1 core, how many puma workers, threads and what database pool size is appropriate?
What's the general thumb here?
- 812
12
votes
2
answers
9k
views
Does Sidekiq execute jobs in the order they are sent to a worker?
I have a rake task which is going to call 4 more rake tasks, in order:
rake:one
rake:two
rake:three
rake:four
Rake tasks one, two, and three are getting data and adding it to my database. Then rake:...
- 5,513
12
votes
5
answers
4k
views
TCP Socket communication between processes on Heroku worker dyno
I'd like to know how to communicate between processes on a Heroku worker dyno.
We want a Resque worker to read off a queue and send the data to another process running on the same dyno. The "other ...
- 6,442
12
votes
1
answer
2k
views
How to get supervisord to restart hung workers?
I have a number of Python workers managed by supervisord that should continuously print to stdout (after each completed task) if they are working properly. However, they tend to hang, and we've had ...
- 2,271
12
votes
1
answer
3k
views
What's best practice for HA gearman job servers
From gearman's main page, they mention running with multiple job servers so if a job server dies, the clients can pick up a new job server. Given the statement and diagram below, it seems that the job ...
- 4,031
11
votes
3
answers
9k
views
Node.js sending an object with function definitions to worker thread
So I am working on a project in Node.js and I want to open up some extra threads to handle the processing load more efficiently. But I am using classes with function definitions with them and when I ...
- 131
11
votes
1
answer
12k
views
Airflow: what do `airflow webserver`, `airflow scheduler` and `airflow worker` exactly do?
I've been working with Airflow for a while now, which was set up by a colleague. Lately I run into several errors, which require me to more in dept know how to fix certain things within Airflow.
I do ...
- 273
11
votes
1
answer
2k
views
What's the difference between `retry_after` and `timeout` options for Laravel queues?
I came across an issue in my code base where a job was being terminated early while its timeout was set to 540 seconds. However after looking in the documentation I noticed it was an issue with the ...
- 7,213
11
votes
0
answers
2k
views
The call in the CallbackToFutureAdapter is executed the second time the project is rebuild
I'm having a problem with ListenableFuture, here is the whole code
MainActivity.java:
package com.example.app16;
import android.os.Bundle;
import android.view.View;
import androidx.appcompat.app....
- 141
10
votes
2
answers
25k
views
What is a "worker" in Odoo?
What is the purpose of workers? Are these workers for multi-threading or something else? When the Odoo instance start I see at least 6 workers on command line that informed:
2016-03-10 13:55:09,602 ...
- 608
10
votes
6
answers
9k
views
How to check in C++ if the system is active?
I'm writing code that need to run only when there is no human activity on the PC, like when the screensaver is running. Any suggestions on how to do this in c++ under windows?
@talnicolas, simply to ...
- 259
10
votes
1
answer
3k
views
Elastic Beanstalk Worker's SQS daemon getting 504 gateway timeout after 1 minute
I have an Elastic Beanstalk worker that can only run one task at a time and it takes some time to do so (from a few minutes to, hopefully, less than 30 minutes), so I'm queuing my tasks on a SQS.
On ...
- 2,077
10
votes
3
answers
2k
views
How to share worker among two different applications on heroku?
I have two separate applications running on heroku and pointing to same database, first one responsible for user interfaceand second one for admin interface, I am using sidekiq with redis for ...
- 7,207
10
votes
2
answers
3k
views
Differentiate driver code and work code in Apache Spark
In Apache Spark program, how do we know which part of code will execute in driver program and which part of code will execute in worker nodes?
- 345
10
votes
2
answers
9k
views
Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
mp.set_start_method('spawn')
total_count = Counter(0)
pool = mp.Pool(initializer=init, initargs=(total_count,), processes=num_proc)
pool.map(part_crack_helper, product(seed_str, repeat=4))
pool....
- 1,634
10
votes
2
answers
5k
views
Would starting APScheduler in a uwsgi app end up with one scheduler for each worker?
I have a flask application in which I need the scheduling feature of APScheduler. The question is:
Where do I start the scheduler instance?
I use uwsgi+nginx to serve this application with multiple ...
- 2,009
10
votes
2
answers
3k
views
How to make each unicorn worker of my Rails application log to a different file?
How can I make each unicorn worker of my Rails application writting in a different log file ?
The why : problem of mixed log files...
In its default configuration, Rails will write its log messages ...
- 101
10
votes
1
answer
3k
views
Is it feasible to run multiple processeses on a Heroku dyno?
I am aware of the memory limitations of the Heroku platform, and I know that it is far more scalable to separate an app into web and worker dynos. However, I still would like to run asynchronous tasks ...
- 271
9
votes
3
answers
7k
views
gunicorn doesn't process simultaneous requests concurrently
I am trying to serve long running requests using gunicorn and its async workers but I can't find any examples that I can get to work. I used the example here but tweaked to add a fake delay (sleep for ...
- 469
9
votes
1
answer
15k
views
.NET Core 3 Worker Service Settings Dependency Injection
I have the following successfully working in .NET Core 2, but it doesn't work in .NET Core 3.
I've made a new Worker Service .Net Core 3 project from scratch and added only the minimum to recreate my ...
- 490
9
votes
1
answer
2k
views
is Magento thread-safe?
Is anyone using Magento with Apache worker MPM?
I've read conflicting reports of stability and security using worker instead of prefork, but have also read that worker is much faster than prefork.
- 641
9
votes
2
answers
11k
views
Right way to start a worker thread with Quarkus?
I am implementing a Quarkus server. On server start, a (never ending) background process should be started.
I know I can observe the start event with an @ApplicationScopedbean which implements:
void ...
- 237
9
votes
1
answer
9k
views
Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': Value at index 0 does not have a transferable type
This is the code of my worker:
onmessage = function (event) {
postMessage({'data': event.data}, ['http://localhost:9000']);
};
And this is the code where I use the worker:
var worker = new ...
- 174
9
votes
1
answer
5k
views
Heroku workers for node.js
I am starting with Heroku and I have a webapp that has a part that needs to run once every week (Mondays preferably). I had been reading something about workers: here and here and here... But I still ...
- 1,372