Questions tagged [spring-batch]
Spring Batch is a lightweight, comprehensive framework aimed at enabling the development of batch applications vital for the daily operations of enterprise systems. Batch applications in this context refers to automated offline systems targeted towards bulk data processing.
3,620
questions with no upvoted or accepted answers
10
votes
1
answer
5k
views
JMS and Spring batch
Our project is to integrate two applications, using rest api of each, using JMS(to provide asynchronous nature) and spring batch to read bulk data from the JMS queue and process it and then post it ...
- 119
8
votes
0
answers
4k
views
Spring Batch + Spring Boot - shutdown process
I'm new to Spring Batch and I can't figure out how to terminate the Spring Boot process after job is finished.
In my BatchConfiguration class i configure jobs and steps:
@Configuration
@...
- 1,211
7
votes
1
answer
2k
views
Spring Batch Job taking previous execution parameters
I am using spring cloud dataflow and have created a spring cloud task which contains a job. This job has a parameter called last_modified_date, which is optional. In the code, I have specified which ...
- 313
7
votes
0
answers
2k
views
Spring batch skips records
I have a spring batch (3.0.4.RELEASE) job configured to index some records in solr. I'm not experiencing any exceptions and the job responds as completing successfully, however on occasion it is only ...
- 193
6
votes
1
answer
6k
views
Spring-Batch: Writing objects to lines with fixed length?
Spring-Batch provides the class FixedLengthTokenizer which makes it easy to read different offsets of a single line into the fileds of an object. Whereby the content of each field is extracted from ...
- 3,627
6
votes
0
answers
966
views
Spring-integration-ftp polls file even when my application is stopped
As a follow up to the question -
Same file gets picked up again and again in spring-ftp but with different names
I have the following configuration in my application.xml
<?xml version="1.0" ...
- 1,076
6
votes
0
answers
2k
views
Make other components aware of @WebIntegrationTest random port
I've been playing around with Spring boot these last days, and I really like the @WebIntegrationTest annotation, that allows to startup a Tomcat and deploy a Spring boot app in it automatically. It ...
- 6,943
6
votes
1
answer
2k
views
How do I set up to run batch jobs based on priority?
I have looked through documentation as well as made numerous Google attempts to learn, but so far I've come up empty-handed:
1) Is there a settable parameter that would allow me to achieve the ...
- 61
6
votes
0
answers
7k
views
Exception : HibernateJpaDialect does not support custom isolation levels due to limitations in standard JPA
i am relatively new to hibernate and spring batch. I am trying to configure similar db configurations for both my application and job-repository (in spring batch). It's a JPA + Hibernate + Spring ...
- 395
6
votes
0
answers
13k
views
Spring batch JMS writer/reader example
Anybody know of a good resource for a detailed (more so than the Spring Batch docs) look at the uses of JMS Item Writer/Reader in Spring Batch?
Specifically, and because I'm being tasked with trying ...
- 1,025
5
votes
0
answers
305
views
Spring batch step completing but Job not ending
I have a Job that has only one step which contains JdbcPagingItemReader,Custom Processor and custom writer that writes to elasticsearch.
Step configuration is
jobBuilderFactory.get("job")
.&...
- 105
5
votes
0
answers
1k
views
Is there a way to resolve OptimisticLockingFailureException in Spring Batch?
I have a Spring Batch configuration as such.
@Bean(BatchJobConstant.WITHDRAW_BATCH_JOB)
public Job importJob(@Qualifier(BatchJobConstant.CUSTOM_READER) CustomListItemReader<WithdrawDetails> ...
- 171
5
votes
1
answer
4k
views
How to process a large file from S3 and use it in spring batch
I've CSV file which contains millions of records and the size will be around 2GB. My use case here is to read the CSV file from S3 and process it. Please find my code below:
In the below code, I'm ...
- 397
5
votes
1
answer
599
views
Spring batch JDBCPagingItemReader, ThreadPoolTaskExecutor and multiple pods in OpenShift
We have a Spring batch application with chunk processing which reads records from DB, processes them and calls a service/performs a few inserts/updates a few tables in the DB as part of writer.
...
- 1,348
5
votes
0
answers
1k
views
Spring batch class loaders conflict issue
I'm trying to test xml reader for spring-batch, but I've faced with the next issue:
java.lang.ClassCastException: class org.vl.batch.domain.CustomerXml cannot be cast to class org.vl.batch.domain....
- 7,285
5
votes
0
answers
2k
views
Spring batch + JPA Persistance exception handling
I am working in a spring batch application which writes to a database using JPA Persistence. I have a scenario, where I have to skip the exceptions during write and the job has to continue without ...
- 351
5
votes
1
answer
3k
views
Spring Batch Auto Configuration with postgres
I am trying a plain Spring batch app using PosgreSQL as the database.
pom.xml:-
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-...
- 7,633
5
votes
0
answers
2k
views
Spring Batch: Retry chunk in smaller chunks vs single item processing
I have an application that uses Spring Batch 3.0.6. The Spring Batch job uses chunk processing. The commit interval is set to 100 and the retry limit is set to 5.
The writer writes a data chunk (...
- 71
5
votes
1
answer
3k
views
looping in Spring batch
I have this simple configuration in Spring Batch for the first step:
<batch:job id="collaborationJob" >
<batch:step id="collaborationJobStep1">
<batch:...
5
votes
0
answers
3k
views
Memory leak with Spring-Batch
We have a spring-batch job responsible for loading test scenarios from a flat file into DB, running a program that we want to test, verifying its results.
We check many scenarios for the executions ...
- 289
5
votes
0
answers
4k
views
In Case of custom Exception Set Step Execution Status other than ABANDONED
I have situation where step can throw custom exception and the execution of job stops on exception. The problem I'm facing is, on exception in 'step2', Spring Batch set step status 'ABANDONED', I ...
- 95
4
votes
1
answer
1k
views
Spring batch restart job failing with transaction exception
I'm testing my spring batch job with restart feature, I'm processing total of nine records with chunk size 5, after processing the first chunk - I'm intentionally failing the second chunk to test the ...
- 1,669
4
votes
1
answer
2k
views
Is there a SPRING batch equivalent in .NET
I came across Spring BATCH and it seems to be a fully fledged batch library. I however was not able to find any .NET equivalent library. There is Summer Batch but it does not seem to be kept up to ...
- 41
4
votes
0
answers
1k
views
SQL Server returned an incomplete response. The connection has been closed.; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException:
We are in the process of moving to Azure SQL Server from Oracle DB for our Spring Batch application that runs for few hours with 50 Threads.
Below is my JDBC Configuration
MSSQL_DRIVERCLASSNAME: "...
- 258
4
votes
0
answers
2k
views
Spring batch meta-data tables in DynamoDB
I have a spring-boot batch application.
Where I'm pulling data from redshift to dump in elastic search.
And I also have a dynamoDB available for the application.
Now, my question is:
is it possible ...
- 1,362
4
votes
0
answers
284
views
How to keep track of process status of nested entities
I'm working on setting up a Job which processes the following data structures:
class A { long id; List<B> bList; }
class B { long id; list<C> cList; }
class C { long id; }
As of now, I'...
- 2,504
4
votes
2
answers
5k
views
How do I add parameters to a Spring Batch item reader?
I feel like this is a really basic thing, but I can't find docs on how to do it. All the documentation/examples I'm finding assumes static queries. I can do this as a static query, but I want to know ...
- 3,843
4
votes
0
answers
3k
views
SpringBatch - Execute next step even on failure of first step
I am looking for
Execute steps sequentially
Irrespective of Failure or Success status of earlier step next step should be executed
Below is the code snippet which helps to achieve point 1), but job ...
- 107
4
votes
2
answers
2k
views
How to externalize application properties in Spring Cloud Data Flow taks
I currently have a regular SpringBoot App that I'm splitting into 2 applications in order to gain some of the benefits from Spring Cloud Dataflow and Spring Batch:
A SpringBoot app responsible for ...
- 163
4
votes
1
answer
3k
views
Spring batch paginated reader and Exception handling
We created a custom item reader which extends AbstractPaginatedDataItemReader. Spring-batch allows to manage which exception stops or not a job (skipped exceptions).
In "classic" spring-batch readers,...
- 41
4
votes
0
answers
3k
views
Spring Batch how to remove duplicate items in processor where last entry have preference before send to item writer
I read a flat file (for example a .csv file with 1 line per User, Ex: UserId,age,rate,marks).
Below is example
user1,27,good,1111111
user2,56,fair,222222
user1,28,good,333333
So as per above data ...
4
votes
0
answers
2k
views
How to run parallel jobs in spring batch
I want to run jobs in parallel. I am using Java configuration.
I have tried many solutions from here:-
using spring batch to execute jobs in parallel
How to run spring batch jobs in parallel
I have ...
- 139
4
votes
1
answer
4k
views
Spring boot ( spring batch) configuration disable the automatic creation of the database
I am using Spring-boot with Spring-Batch, I have datasource config in my application.properties, but i don't want spring batch to create tables in my database, so for that I use the following ...
- 215
4
votes
0
answers
2k
views
Stop Spring Batch from logging job parameters
We are using Spring batch to automate some processes, and are passing in a number of parameters, some of which are sensitive passwords and can't be logged.
I have prevented these password from being ...
- 55.1k
4
votes
0
answers
963
views
Spring batch with scoped beans and AllowBeanDefinitionOverriding=false
I want to use scoped spring batch beans
<bean id="processor"
class="com.example.Processor"
p:type="#{jobParameters['type']}"
scope="step"/>
and AllowBeanDefinitionOverriding = false
...
- 7,354
4
votes
1
answer
873
views
Spring Batch - Is there a way to execute a TaskletStep asynchronously?
I want to trigger an external system process from spring batch TaskletStep and register a callback. When the external process is complete it will signal my original spring batch job and the job will ...
- 113
4
votes
0
answers
3k
views
Spring Batch suitable ItemReader when item resource is rest api
I have a question
My datasource is rest API such as, I call this api
localhost:8080/datawarehouse/phone?size=100&order=name
the api result List,
{
[
{"name":"A","number":"12-333-3333"},
{...
- 165
4
votes
0
answers
702
views
Get list of executions filtered by parameter value
I am using Spring-batch 3.0.4 stable. While submitting a job I add some specific parameters to its execution, say, a tag. Jobs information is persisted in the DB.
Later on I will need to retrieve all ...
- 1,081
4
votes
0
answers
10k
views
OptimisticLockingFailureException updating step execution after commit failure in Spring Batch
I am performing some integration testing for my application in which I am trying to force an exception whenever there is an update/insert happening in the database. Whenever any exception thrown due ...
- 59
4
votes
1
answer
1k
views
using spring batch to execute jobs in parallel
I have a use case as follows:
1)There is a Parentjob which has multiple child jobs.
2)All child jobs should be executed in parallel.
3)The parent job should wait until all child jobs are done.
4)One ...
- 19.1k
4
votes
2
answers
2k
views
JdbcBatchItemWriter processing on tables having FK relation
I have a requirement to insert data into two tables using JdbcBatchItemWriter.
I have two tables TABLE_A, TABLE_B.
TABLE_B is child of TABLE_A with Foreign Key Relation. I am successful on ...
- 845
4
votes
2
answers
9k
views
Spring Batch Item Processor not Executing
I am creating a Spring Batch job using the Spring Batch template included with STS 2.81 and using examples from Spring Batch in Action by Manning. I am able to execute the chunk reader and writer ...
- 41
4
votes
1
answer
7k
views
How to use StaxEventItemReader in Spring Batch for complex xml
Please help me in writing the ItemReader()
<File>
<FileDate>05/28/2016</FileDate>
<RecordCount>75</RecordCount>
<Transaction>
...
- 89
4
votes
2
answers
2k
views
Append data in spring batch
I'd like to know how to specify that I want the data output by the second job being appended to the data output by the first job:
job1 ---> data 1 ---> Output XML file
job2 ---> data 2 ---> SAME ...
- 796
3
votes
2
answers
2k
views
How to automatically delete successful Spring Batch jobs from the DB
We have a Spring Batch application that runs millions of batches every month. The vast majority of the jobs are successful and only a handful of them fail and require attention.
We have no actual need ...
- 6,716
3
votes
0
answers
106
views
Azure Kubernetes : Spring Batch running on Ubuntu 18.04 is hanging
I have a Spring batch 2.1.x application deployed on Azure Kubernetes. Base image is Ubuntu 18.04. I see that the process is hanging at times.
Process flow:
8119 : kubectl command line command to ...
- 258
3
votes
1
answer
2k
views
Can We Process Whole Chunk Together in Spring Batch Processor
I have a scenario where I have million records in employee staging table, I need to enrich the value and store it in employee final table. Now I am using chunk processing with chunk size of 10,000.
In ...
- 73
3
votes
0
answers
157
views
Spring Cloud Data Flow global properties for Composed Tasks
I am trying to pass global properties into Composed tasks. When I specify each and every job, it works, but there is configuration duplicity. I tried multiple approaches, but the properties are not ...
- 2,115
3
votes
1
answer
2k
views
Spring Not a Managed Type class
Note: I`m not using spring boot
All the errors in here
Error creating bean with name 'writeData': Unsatisfied dependency
expressed through field 'userRepository'; nested exception is
org....
- 46
3
votes
0
answers
381
views
Spring Batch with Spring Cloud Data Flow running out of DB connections
We have Spring Cloud Data Flow running in Kubernetes in order to orchestrate Spring Batch jobs. For each new file we have in, Spring Cloud Data Flow spins up a new Spring Batch task.
Spring Batch ...
- 1,086