Questions tagged [spring-batch]
Spring Batch is a lightweight, comprehensive framework aimed at enabling the development of batch applications vital for the daily operations of enterprise systems. Batch applications in this context refers to automated offline systems targeted towards bulk data processing.
8,532
questions
358
votes
52
answers
1.2m
views
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured
I am working on a Spring Boot Batch example with MongoDB and I have already started the mongod server.
When I launch my application, I am getting the error below.
Any pointers for this issue?
******...
- 8,646
93
votes
12
answers
149k
views
How can we share data between the different steps of a Job in Spring Batch?
Digging into Spring Batch, I'd like to know as to How can we share data between the different steps of a Job?
Can we use JobRepository for this? If yes, how can we do that?
Is there any other way of ...
- 1,063
86
votes
31
answers
380k
views
Parameter 0 of constructor in required a bean of type 'java.lang.String' that could not be found
I am working on spring batch with spring boot 2.X application, actually its existing code i am checked out from git. While running the application it fails due to below error only for me and same code ...
- 1,033
84
votes
11
answers
181k
views
How to get access to job parameters from ItemReader, in Spring Batch?
This is part of my job.xml:
<job id="foo" job-repository="job-repository">
<step id="bar">
<tasklet transaction-manager="transaction-manager">
<chunk commit-interval="...
- 103k
65
votes
10
answers
118k
views
Spring-Batch without persisting metadata to database?
I want to create a spring-batch job, but I want to run it without any database persistence. Unfortunately spring-batch requires to write metadata ob the job cycles to a database somehow, thus procing ...
- 84.2k
55
votes
12
answers
141k
views
Spring Batch Framework - Auto create Batch Table
I just created a batch job using Spring Batch framework, but I don't have Database privileges to run CREATE SQL. When I try to run the batch job I hit the error while the framework tried to create ...
- 691
53
votes
6
answers
64k
views
Use of multiple DataSources in Spring Batch
I am trying to configure a couple of datasources within Spring Batch. On startup, Spring Batch is throwing the following exception:
To use the default BatchConfigurer the context must contain no ...
- 739
48
votes
8
answers
39k
views
How can I get started with Spring Batch? [closed]
I'm trying to learn Spring Batch, but the startup guide is very confusing. Comments like
You can get a pretty good idea about
how to set up a job by examining the
unit tests in the
org....
- 31.5k
48
votes
9
answers
37k
views
What's the equivalent of Python's Celery project for Java?
I am trying to find an equivalent of Celery project for Java environment, I have looked at Spring Batch, but are there any better alternatives for distributed task queues.
Thanks.
- 1,475
48
votes
7
answers
89k
views
How to java-configure separate datasources for spring batch data and business data? Should I even do it?
My main job does only read operations and the other one does some writing but on MyISAM engine which ignores transactions, so I wouldn't require necessarily transaction support. How can I configure ...
- 801
44
votes
2
answers
36k
views
How Spring Boot run batch jobs
I followed this sample for Spring Batch with Boot.
When you run the main method the job is executed.
This way I can't figure out how one can control the job execution. For example how you schedule a ...
- 22.3k
41
votes
8
answers
47k
views
javax.management.InstanceNotFoundException: org.springframework.boot:type=Admin,name=SpringApplication
I developed a Spring boot application and the functionality of the application is working fine. However, at the time of startup, I see the below exception multiple times.
When I ran the same ...
- 5,901
41
votes
6
answers
57k
views
Multiple itemwriters in Spring batch
I am currently writing a Spring batch where I am reading a chunk of data, processing it and then I wish to pass this data to 2 writers. One writer would simply update the database whereas the second ...
- 3,194
39
votes
9
answers
98k
views
Why Spring's jdbcTemplate.batchUpdate() so slow?
I'm trying to find the faster way to do batch insert.
I tried to insert several batches with jdbcTemplate.update(String sql), where
sql was builded by StringBuilder and looks like:
INSERT INTO ...
- 1,272
39
votes
1
answer
57k
views
How Does Spring Batch Step Scope Work
I have a requirement where I need to process files based on the rest call in which I get the name of the file, I am adding it to the job parameter and using it while creating the beans.
I am ...
- 393
38
votes
1
answer
24k
views
Advantages of spring batch [closed]
I understood that spring batch framework processes data in chunks. However, I was thinking that when the same chunking functionality can be acheived through java why do we need to go for batch ...
- 845
35
votes
2
answers
38k
views
Spring-batch @BeforeStep does not work with @StepScope
I'm using Spring Batch version 2.2.4.RELEASE
I tried to write a simple example with stateful ItemReader, ItemProcessor and ItemWriter beans.
public class StatefulItemReader implements ItemReader<...
- 1,993
34
votes
1
answer
37k
views
Difference between Step, Tasklet and Chunk in Spring Batch [closed]
What is the difference between Step, Tasklet and Chunk in spring batch.?
Also, How to execute step in parallel via Spring Batch. ?
- 407
33
votes
7
answers
26k
views
I am getting error Table 'test.batch_job_instance' doesn't exist
I am new to Spring Batch. I have configured my job with inmemoryrepository. But still, it seems it is using DB to persist job Metadata.
My spring batch Configuration is :
@Configuration
public class ...
- 520
33
votes
5
answers
46k
views
Spring Batch: One reader, multiple processors and writers
In Spring batch I need to pass the items read by an ItemReader to two different processors and writer. What I'm trying to achieve is that...
+---> ItemProcessor#1 ---> ...
- 4,586
32
votes
8
answers
32k
views
Spring Batch: org.springframework.batch.item.ReaderNotOpenException: Reader must be open before it can be read
I read SO related questions but the solutions don't work for me.
I get the org.springframework.batch.item.ReaderNotOpenException: Reader must be open before it can be read exception.
Below is my ...
- 1,296
32
votes
6
answers
155k
views
Could not open JPA EntityManager for transaction; nested exception is java.lang.IllegalStateException
I am quite new to Spring and Spring-Batch in particular.
Still I somehow managed to install the Spring Batch-Admin. I added custom jobs and Hibernate/JPA for persistence.
Everything is working as ...
- 1,888
32
votes
10
answers
62k
views
Spring Batch ORA-08177: can't serialize access for this transaction when running single job, SERIALIZED isolation level
I am getting this exception with SERIALIZED isolation level on JobRepository in Spring Batch:
org.springframework.dao.CannotSerializeTransactionException: PreparedStatementCallback; SQL [INSERT into ...
- 2,344
31
votes
4
answers
98k
views
How to get JobParameter and JobExecutionContext in the ItemWriter?
I want to retrieve JobParameter and JobExecutionContext object in my ItemWriter class.
How to proceed?
I tried implementing StepExecutionListener through which I am just calling the parent class ...
- 329
28
votes
2
answers
82k
views
Spring batch Cron expression: to run every 3 hours
I want my spring batch job to run every 3 hours
I used expression * * */3 * * ? this starts the job at the hour that is divisible by 3 e.g. say the server was started at 2 PM the job starts executing ...
- 381
28
votes
1
answer
47k
views
Why does destroy method 'close' fail for JPAPagingItemReader configured with Java config?
We are trying to convert our Spring-Batch jobs from XML configuration to Java configuration. We are using Spring 4.0.1.RELEASE and Spring Batch 2.2.1.RELEASE.
After converting one job, the following ...
- 14.7k
27
votes
5
answers
42k
views
Triggering spark jobs with REST
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my ...
- 1,963
26
votes
3
answers
59k
views
How do I set JobParameters in spring batch with spring-boot
I followed the guide at http://spring.io/guides/gs/batch-processing/ but it describes a job with no configurable parameters. I'm using Maven to build my project.
I'm porting an existing job that I ...
- 6,812
25
votes
2
answers
89k
views
Storing in JobExecutionContext from tasklet and accessing in another tasklet
I have a requirement in which a tasklet, stores all the files in the directories in an arraylist. The size of the list is stored in the job execution context. Later this count is accessed from another ...
- 12.5k
25
votes
3
answers
34k
views
How to load all files of a folder to a list of Resources in Spring?
I have a folder and want to load all txt files to a list using Spring and wildcards:
By annotation I could do the following:
@Value("classpath*:../../dir/*.txt")
private Resource[] files;
But how ...
- 84.2k
25
votes
1
answer
16k
views
ETL in Java Spring Batch vs Apache Spark Benchmarking
I have been working with Apache Spark + Scala for over 5 years now (Academic and Professional experiences). I always found Spark/Scala to be one of the robust combos for building any kind of Batch or ...
- 863
24
votes
4
answers
32k
views
spring batch vs quartz jobs?
I am new to batch processing. I am trying to start with simple scheduler and job. But i am confused b/w
spring batch vs quartz jobs. My understanding is
Quartz :- quartz provides both frameworks i....
- 10.3k
24
votes
4
answers
25k
views
how to stop spring batch scheduled jobs from running at first time when executing the code?
i'm using spring batch 2.2.4 with quartz to run some jobs at certain time
the problem is the jobs always run after executing the code at the first time then it runs based on the scheduled time. I ...
- 355
23
votes
5
answers
30k
views
How to process logically related rows after ItemReader in SpringBatch?
Scenario
To make it simple, let's suppose I have an ItemReader that returns me 25 rows.
The first 10 rows belong to student A
The next 5 belong to student B
and the 10 remaining belong to student C
...
- 3,001
22
votes
3
answers
41k
views
run spring batch job from the controller
I am trying to run my batch job from a controller. It will be either fired up by a cron job or by accessing a specific link.
I am using Spring Boot, no XML just annotations.
In my current setting I ...
- 461
22
votes
4
answers
31k
views
Return multiple items from spring batch ItemProcessor
I'm writing a spring batch job and in one of my step I have the following code for the processor:
@Component
public class SubscriberProcessor implements ItemProcessor<NewsletterSubscriber, Account&...
- 19.1k
21
votes
7
answers
35k
views
Spring batch scope issue while using spring boot
I'm having Standalone spring batch job. This works perfectly fine when in JUNIT
@RunWith(SpringJUnit4ClassRunner.class)
//@SpringApplicationConfiguration(classes = KPBootApplication.class)
@...
- 9,842
20
votes
3
answers
17k
views
How to get an ideal number of threads in parallel programs in Java?
I need to get an ideal number of threads in a batch program, which runs in batch framework supporting parallel mode, like parallel step in Spring Batch.
As far as I know, it is not good that there ...
- 1,245
20
votes
1
answer
8k
views
BeanDefinitionParsingException: Configuration: The element [step2] is unreachable
I had spring batch job similar to this one:
<batch:job id="job">
<batch:step id="step1">
...
</batch:step>
<batch:step id="step2">
...
</...
- 10.4k
20
votes
1
answer
19k
views
Need understanding of spring.handlers and spring.schemas
I have some questions derived from a problem that I have already solved through this other question. However, I am still wondering about the root cause. My questions are as follows:
What is the ...
- 3,001
20
votes
1
answer
33k
views
How does Spring Batch transaction management work?
I'm trying to understand how Spring Batch does transaction management. This is not a technical question but more of conceptual one: what approach does Spring Batch use and what are the consequences of ...
- 665
19
votes
4
answers
41k
views
Spring batch jpaPagingItemReader why some rows are not read?
I 'm using Spring Batch(3.0.1.RELEASE) / JPA and an HSQLBD server database.
I need to browse an entire table (using paging) and update items (one by one). So I used a jpaPagingItemReader. But when I ...
- 1,804
19
votes
1
answer
15k
views
Difference between Batch Status and Exit Status in Spring Batch
Difference between Batch Status and Exit Status in Spring Batch
- 253
19
votes
1
answer
55k
views
Add parameter to job context from tasklet step and use in later steps in Spring Batch
For now, I'm using jobParameters to get the filenames for both my FlatFileItemReader and FlatFileItemWriter. It's okay for testing my batch, but my goal is to read a file in some directory (there is ...
- 1,543
19
votes
2
answers
18k
views
Spring Batch: Which ItemReader implementation to use for high volume & low latency
Use case: Read 10 million rows [10 columns] from database and write to a file (csv format).
Which ItemReader implementation among JdbcCursorItemReader & JdbcPagingItemReader would be suggested? ...
- 757
19
votes
1
answer
23k
views
Spring @Transactional and JDBC autoCommit
On my actual application, I have a DBCP connection pool which doesn't have JDBC autoCommit=false set. It seems to have the default autoCommit=true.
This is probably a mistake but I'd like to ...
- 91.1k
18
votes
2
answers
15k
views
Difference between spring batch remote chunking and remote partitioning
What is the difference between spring batch remote chunking and remote partitioning?
I can not understand the difference between remote chunking and remote partitioning in spring batch. Could anybody ...
- 3,352
18
votes
1
answer
15k
views
Best Spring batch scaling strategy
We have simple batch processes which are working fine. Recently we have new reqmnt to implement new batch process to generate reports. We have diff source of data to read to prepare this reports. ...
- 2,961
18
votes
2
answers
19k
views
What is the difference between spring scheduled tasks and spring batch jobs
I dont understand the difference between scheduled tasks and batch jobs in spring.
By scheduled tasks I mean the ones which are configured like these:
@EnableScheduling
public class AppConfig{
..
...
- 313
18
votes
1
answer
32k
views
Deciding between Spring Batch Step, Tasklet or Chunks
I have a straight forward requirement in which, i need to read a list of items(from DB) and need to process the items and once processed, it has to be updated into DB.
I'm thinking of using Spring ...
- 191