Questions tagged [spring-data-hadoop]
Spring for Apache Hadoop is an open-source project that provides unified configuration model and easy to use APIs for using HDFS, MapReduce, Pig, and Hive as well as developing and deploying YARN applications.
47
questions
27
votes
5
answers
42k
views
Triggering spark jobs with REST
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my ...
- 1,963
7
votes
2
answers
8k
views
How to submit multiple Spark applications in parallel without spawning separate JVMs?
The problem is that you need to launch separate JVM to create separate session with different number of RAM per job.
How to submit few Spark applications simultaneously without manually spawning ...
- 45.3k
7
votes
0
answers
1k
views
Why is spring-hadoop end-of-lifed and is there a successor?
According to the README in spring-hadoop, the project has reached an End-Of-Life:
NOTICE: The Spring for Apache Hadoop project has reached End-Of-Life
status on April 5th, 2019. The final Spring for ...
- 32k
4
votes
2
answers
12k
views
Use Hbase with spring boot [duplicate]
Is there a sample project show how to use hbase in spring boot ?
I read this link How to use hbase with Spring Boot using Java instead of XML?. But the instruction is not detail enough.
- 195
4
votes
2
answers
3k
views
maven-shade-plugin vs maven-assembly-plugin
I wrote a hadoop MR job using spring-data-hadoop and used maven-shade-plugin to pack all necessary jars.
But when I ran the job from comand line:hadoop jar , it gave me Exception for ClassNotFound ...
- 81
4
votes
0
answers
530
views
Spring Hadoop Java config
What is the current state of JavaConfig support? As I see it's possible to set up Hadoop configuration this way: http://docs.spring.io/spring-hadoop/docs/2.2.0.RELEASE/reference/html/springandhadoop-...
3
votes
1
answer
3k
views
Spring Boot Yarn - Passing Command line arguments
i'm trying to pass command line arguments in my Spring Boot Yarn application and am having difficulties. i understand that i can set these in the yml document spring.yarn.appmaster.launchcontext....
- 2,137
3
votes
1
answer
1k
views
trap Spring context close error
I have a spring Integration project using spring-hadoop.
I have an Hbase template as follows,
<beans:bean id="fsh" class="org.springframework.data.hadoop.fs.FsShell">
</beans:bean>
<...
- 1,510
3
votes
0
answers
163
views
Spring for Hadoop HBase
I have a question to Spring community.
As stated in the article, Spring for Hadoop will no longer be supported on April, 2019.
Is (or will) there be any other Spring project for Hadoop, HBase (...
- 444
2
votes
1
answer
2k
views
Unknown host exception when using spring data hadoop to connect to Cloudera QuickStart VM Hbase
I use QuickStart VMs for CDH 5.3.x
I try to implement this spring hadoop sample for hbase
The sample from the host computer will connect to Hbase in the VM to create table, add data, read data.
...
- 195
2
votes
1
answer
2k
views
Spring Hadoop config - No qualifying bean of type org.apache.hadoop.conf.Configuration
I am trying to configure beans for Hadoop/Hive environment. According to documentation I need Apache Hadoop Configuration class, which should be autowired. See: http://docs.spring.io/spring-hadoop/...
- 4,158
2
votes
2
answers
3k
views
spring data hive integration with hive template
I am trying to use spring data hadoop to integrate hive into my application and running into some issues. First thing I am not sure about is <hdp:hive-server host="some-other-host" port="10001" /&...
- 2,401
1
vote
1
answer
4k
views
Spring data - hadoop connectivity
I'm trying out Spring Data - Hadoop for executing the MR code on a remote cluster from my local machine's IDE
Hadoop 1.1.2, Spring 3.2.4, Spring-Data-Hadoop 1.0.0
My bean configuration file viz. ...
- 3,572
1
vote
1
answer
439
views
How to "findAllBy{Column}" with HbaseTemplate?
I have been developing my first RESTful server utilizing such technologies as Spring and HBase.
Message below is the core model of my server;
@AllArgsConstructor
@Getter
public class Message {
...
- 1,862
1
vote
1
answer
2k
views
Spring Hadoop (Hive)
Am I supposed to understand how to implement Spring Hadoop / Hive by reading this quick guide? I don't know where the xml configurations should go (maybe hive.xml)?
https://docs.spring.io/spring-...
- 99
1
vote
1
answer
2k
views
spring hbaseTemplate throws java.lang.IllegalArgumentException: Not a host:port pair: PBUF
I am a newbie for Hbase and I want to continue to use spring solution, hBaseTemplate to access HBase. But I tested a lot of times and can never be successful in doing so. This is what I did.
The ...
- 3,433
1
vote
1
answer
935
views
how to pass parameters from web requests to spring boot yarn application
I'm using spring-boot and spring-boot-yarn to submit yarn applications to a cluster.
My use-case is close to the one described in this tutorial https://github.com/spring-guides/gs-yarn-basic.
The ...
1
vote
1
answer
948
views
Spring HbaseTemplate keeping connection alive
I managed to integrate Hbase into a Spring app using HbaseTemplate:
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.hadoop.hbase.HbaseTemplate;
import ...
- 51.3k
1
vote
0
answers
140
views
Fix Avro filename with spring data hadoop
My goal is to write a directory on HDFS. For that I use:
spring-data-hadoop:2.4.0.RELEASE
spring-data-hadoop-store:2.4.0.RELEASE
In my config class I define a bean
@Bean
public DataStoreWriter<...
- 1,234
1
vote
0
answers
359
views
spring distcp creates target folder as filename
I am using spring DistCp to copy a file within hdfs. My code looks like this
distcp.copy(null, null, null, "/tmp", null, null, null, null, null, null, null,
new String[]{"/user/aq728y/...
- 2,401
1
vote
1
answer
407
views
running a distcp job from spring in hadoop 2.x
I have been using spring data hadoop in on of my projects and have been able to run distcp jobs in hadoop 1.x. Recently we have upgraded to hadoop 2.x and for that I upgraded spring-data-hadoop to 2.0....
- 2,401
1
vote
0
answers
362
views
Spring Data Hadoop
I using Hadoop 2.4.x, Spring 4.0.6, Spring-Data-Hadoop 2.0.1.RELEASE-hadoop24
I'm running only jar file then no problems. But Running with tomcat have some error;
my hadoop config is below:
<...
- 141
1
vote
0
answers
102
views
Is jar creation necessary to execute MR on a remote cluster
I have been trying Spring Data for Hadoop to execute a MR job from my local Windows STS on a remote Hadoop cluster.
The issue I face is mentioned in detail here
There's a similar thread that has ...
- 3,572
1
vote
4
answers
2k
views
ClassNotFoundException after job submission
I'm trying out Spring Data - Hadoop for executing the MR code on a remote cluster from my local machine's IDE
//Hadoop 1.1.2, Spring 3.2.4, Spring-Data-Hadoop 1.0.0
Tried with these versions :
...
- 3,572
0
votes
2
answers
67
views
Issue when executing spring bean
I've a bean named textFileWriter to write string entities to HDFS. I've configured the spring bean in bean config file. While executing am getting NullPointerException. Please help me on this.
My ...
- 1,675
0
votes
1
answer
177
views
Spring Yarn @OnContainerStart - how to invoke Mapper?
i'm using the Spring Yarn package with Spring Boot and i'm trying to figure out how i can start a Mapper from the @OnContainerStart event. how do i pass arguments to the mapper? how do i configure ...
- 2,137
0
votes
1
answer
2k
views
Issue when writing to HDFS using spring data hadoop
I was trying to write a simple text to HDFS using spring data hadoop.
But I'm getting an unknown issue upon writing.
Exception in thread "main"
org.springframework.data.hadoop.store....
- 1,675
0
votes
3
answers
408
views
Customizing Yarn container
I'm testing spring-yarn integration API and I'm little confused about what is the best practice of Yarn container customization in terms of:
1) If I want to use spring-boot-yarn combo, what is the ...
- 291
0
votes
2
answers
409
views
How to filter records according to `timestamp` in Spring Data Hadoop?
I have a hbase table with a sample record as follows:
03af639717ae10eb743253433147e133 column=u:a, timestamp=1434300763147, value=apple
10f3d7f8fe8f25d5bdf52343a2601227 ...
- 398
0
votes
1
answer
601
views
how can i use spring data for apache hadoop for cdh 5.4.x?
The official document note that "Spring for Apache Hadoop 2.2 GA" only support CDH 5.3.3
Is there a way to use "spring data hadoop" for CDH 5.4.X or i must wait for the next version ? Will the 2.3 ...
- 195
0
votes
0
answers
797
views
hadoop configuration error in runtime with openjdk11
I am migrating our application to openjdk11 and with this setup my application is throwing below error.
PLease help on this
Note : With Jdk 1.8 the same code and configurations are working fine .
Java ...
- 44
0
votes
0
answers
179
views
Spring Hadoop - Unable to locate Spring NamespaceHandler for XML schema namespace
I want to enable Hive in my Spring Hadoop project. I understand that we can't use JavaConfiguration so I am using XML. I have an error regarding the Spring Namespace.
Error starting ...
- 99
0
votes
1
answer
602
views
How to simulate hdfs operations using spring data
I'm new to spring data-hadoop and would like to ask one general question. I have files in different format and would like to extract the useful content with Apache Tika and store as text files in HDFS....
- 1,675
0
votes
1
answer
103
views
How to set up hadoop distributed cache using spring data
I'm new to spring data and trying to distribute all the spring data dependencies through distributed cache. But it's not working and no useful resources are found.
My configuration inside application-...
- 1,675
0
votes
1
answer
183
views
Spring boot AMQP and Spring Hadoop together ends up with missing EmbeddedServletContainerFactory bean
I have two small apps, one uses spring-boot-starter-amqp, other uses spring-data-hadoop-boot. I can run them separately without any problems.
When I join them together, app start fails with exception:...
- 4,158
0
votes
1
answer
148
views
Spring Data Hadoop and Hbase 1+
Since recently, hbase released the hbase 1.2, and there is a huge number of bugs fixed in hbase 1.0.0+, I am wondering if Spring is going to upgrade to use the new API.
- 2,102
0
votes
1
answer
221
views
Spring Hadoop Samples Mapreduce throw auxService:mapreduce_shuffle exception
I am following spring hadoop samples here and after trying to run it under mapReduce project:
$ mvn clean package
$ sh ./target/appassembler/bin/wordcount
I got this error:
17:42:55,067 INFO doop....
- 3,433
0
votes
1
answer
710
views
Spring data Hadoop, Hbase Rest API, HBase Java Client : which one would be the best to implement to handle communication between Android and HBase
Does anybody know what is the best method to communicate between HBase database and Android?
Basically I want to do following from my Android app to HBase table:
i. Insert data into it.
ii. Query ...
- 398
0
votes
0
answers
150
views
Can't run a HbaseTemplate demo
I am running HDP2.2 with Hbase 0.98.4.2.2.0.0 and HDFS 2.6.0.2.2.0.0, so I add the 2.1.2.RELEASE version of spring-data-hadoop to my maven pom file, but failed to complie because of "The type org....
- 189
0
votes
1
answer
175
views
Errors while using both Spring Data Hadoop + Spring MVC
I just started with Spring and Hadoop.
I had the following build.gradle which worked fine:
dependencies {
compile 'org.springframework:spring-webmvc:4.0.6.RELEASE'
compile 'jstl:jstl:1.2'
...
- 576
0
votes
1
answer
368
views
Getting Spring-XD and the hdfs sink to work for maprfs
This is a question about spring-xd release 1.0.1 working together with maprfs, which is officially not yet supported. Still I would like to get it to work.
So this is what we did:
1) adjusted the ...
- 3,212
0
votes
1
answer
104
views
Spring jdbchdfs job -
Currently the jdbchdfs job does not have partitionPath for output directory, like the hdfs sink. What is the recommended way for doing it? i don't also see any JIRA for this, any plan for such a thing ...
- 69
0
votes
0
answers
150
views
why "spring-data-hadoop" need to make jar to avoid 'class not found exception'
I have read spring-data-hadoop reference docuements alot.
However, I cannot understand this framework architecture(or usage, necessary)
The most useful point I expected is programming map-reduce job ...
0
votes
0
answers
1k
views
Spring Boot & Hadoop: Protocol message tag had invalid wire type
I'm working on making a web server that will run MapReduce jobs, however I cannot seem to get the server to start. when I try to run the jar for the project, i get the following error
java.io....
0
votes
0
answers
154
views
How to configure MultiOutputFormat from HCatalog API in spring-hadoop project?
I was developing a hadoop application. And now I want to migrate it into a spring-hadoop project. The thing is in my mapreduce job. I was using MultiOutputFormat from the HCatalog API as I want to ...
- 26
0
votes
1
answer
242
views
Issue in running job on Spring-data hadoop
I have created following Mapper and Reducer using Mahout
package mypackage.ItemSimilarity;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org....
- 964
0
votes
1
answer
105
views
How to start a spring data hadoop job later?
I'm new for SpringData Hadoop. I checkout some examples from www, such as [1]:https://github.com/pkainulainen/spring-data-apache-hadoop-examples/tree/master/mapreduce
All of them are configured to ...
- 327