Questions tagged [apache-nifi]
Apache NiFi is an easy to use, powerful, and reliable distributed system to transform and distribute data.
5,258
questions
43
votes
4
answers
39k
views
Difference between Apache NiFi and StreamSets
I am planning to do a class project and was going through few technologies where I can automate or set the flow of data between systems and found that there are couple of them i.e. Apache NiFi and ...
- 1,367
41
votes
1
answer
35k
views
Airbnb Airflow vs Apache Nifi [closed]
Are Airflow and Nifi perform the same job on workflows? What are the pro/con for each one?
I need to read some json files, add more custom metadata to it and put it in a Kafka queue to be processed. I ...
- 1,933
34
votes
3
answers
16k
views
Difference between Nifi and Mini NiFi (MiNiFi)
I just wanted to know how mini NiFi MiNiFi is different from NiFi ?
Do we have any additional feature ? Why there was a need to introduce it?
Any thoughts or link would be great help.
- 2,704
19
votes
1
answer
11k
views
Difference between Apache Beam and Apache Nifi
What are the use cases for Apache Beam and Apache Nifi?
It seems both of them are data flow engines. In case both have similar use case, which of the two is better?
- 354
19
votes
3
answers
27k
views
Difference between kafka and nifi
I'm a beginner with Apache NiFi, but until now All the tutorial that I read speak about the integration of kafka with Nifi. how it kafka is the complementary of Nifi? why we don't use Nifi directly to ...
- 6,211
15
votes
2
answers
12k
views
How to save something that I do on nifi.?
I am new to apache NIFI. I was kind of wondering where is the save button. I tried a sample from a tutorial that I saw on youtube. I would like to save all the processors that I created for future ...
- 257
14
votes
2
answers
24k
views
How does one setup a Distributed Map Cache for NiFi?
I'm brand new to NiFi and simply playing around with processors.
I'm trying to incorporate Wait and Notify processors in my testing, but I have to setup a Distributed Map Cache (server and client?)....
- 4,247
14
votes
3
answers
22k
views
connect SQL to apache nifi
I'm new to nifi and i want to connect SQL server database to nifi and create a data flow with the processors. how can I do this, can any one Help me with this clearly.
Thanks in Advance
sam
- 161
14
votes
3
answers
11k
views
Import Modules in Nifi ExecuteScript
I am new to Nifi and python
i want to execute my python script. So used ExecuteScript and tried to import certain modules. I have imported like this:
import json, sftp, paramiko
Though i have sftp ...
- 911
13
votes
1
answer
16k
views
Python Script using ExecuteStreamCommand
After doing my best to find previous questions and examples relevant to this question, and still not finding the answers that I'm looking for I figured that I would submit a question myself.
...
- 336
12
votes
1
answer
27k
views
How to use NiFi ExecuteScript processor with Python?
I have a very basic setup of the ExecuteScript processor in Apache NiFi with a simple Python script (saved as a .py file) as shown here. In the Properties of the processor, I set the Script Engine to ...
- 1,254
12
votes
2
answers
6k
views
Use Cases of NIFI
I have a question about Nifi and its capabilities as well as the appropriate use case for it.
I've read that Nifi is really aiming to create a space which allows for flow-based processing. After ...
- 6,270
12
votes
2
answers
2k
views
Error during the creating of custom processors apache nifi
I tried to build my package with NiFi custom-processor using mvn clean package command and I got the following output:
[INFO] ------------------------------------------------------------------------
[...
- 121
12
votes
2
answers
4k
views
How Can I View FlowFile or STDOUT of Processor?
When things fail, I'd like to view the flow file, or the output (stdout+stderr) from the problematic processor. Is there an easy way to dump out all of the Flowfile's properties, or to just browse a ...
user5539487
11
votes
1
answer
12k
views
Kafka Avro Consumer with Decoder issues
When I attempted to run Kafka Consumer with Avro over the data with my respective schema,it returns an error of "AvroRuntimeException: Malformed data. Length is negative: -40" . I see others have had ...
- 513
11
votes
1
answer
2k
views
Development Life Cycle for Apache NiFi
I realize that with NiFi, as their doc defines it, "continuous improvement occurs in production". So this doesn’t lend itself to be used as a traditional development tool. However for the project I’m ...
- 801
11
votes
1
answer
25k
views
Creating a Proper avro schema for timestamp record
I would like to know what the proper avro schema would be for some json to avro conversion that is in this format:
{"entryDate": "2018-01-26T12:00:40.930"}
My schema:
{
"type" : "record",
"...
- 786
11
votes
1
answer
12k
views
Apache NiFi ExecuteScript: Groovy script to replace Json values via a mapping file
I am working with Apache NiFi 0.5.1 on a Groovy script to replace incoming Json values with the ones contained in a mapping file. The mapping file looks like this (it is a simple .txt):
Header1;...
- 1,411
10
votes
2
answers
8k
views
APACHE NIFI vs APACHE AIRFLOW vs APACHE FALCON ? Which suits best in the below scenario? [closed]
I am developing a solution in Java which communicates with a set of devices through REST APIs which belongs to different vendors. So for each vendor, there are a set of processes that I have to ...
- 4,651
10
votes
2
answers
3k
views
In Apache NiFi, Is it possible to increase / decrease the scope of a controller?
In Apache NiFi, I created a controller within the appropriate scope at that time. Now I want to use the same controller in other places. My question is can I change the scope to the current processors ...
- 2,568
10
votes
1
answer
10k
views
get response from InvokeHTTP POST
New to NiFi!
I'm using InvokeHTTP nifi processor (using this link as an example to implement my flow http://www.tomaszezula.com/2016/10/30/nifi-and-http-post-configuration/). I currently need to do ...
- 6,270
9
votes
1
answer
10k
views
Apache NIFi MergeContent processor - set demarcator as new line
I want to use MergeContent processor to merge tweets to bulk insert into Elasticsearch index. For this I need command and tweets (each) to be separated by \n.
This is how it should look like
{ ...
- 925
9
votes
1
answer
12k
views
Specify output filename with PutFile processor
Background
I have a workflow that uses a FetchFile processor, some intermediary processors (e.g. RouteOnAttribute, HashContent, etc.), and then a PutFile processor.
Throughout it all, the content is ...
- 4,247
9
votes
1
answer
11k
views
purpose of Funnel in NiFi
In the image, there are 3 GenerateFlowFile processors connected to a LogAttribute processor. I would like to know the need for Funnel. Please clarify.
I understood one such use of funnel where it can ...
- 1,184
9
votes
6
answers
17k
views
PutFile append file
Is there is a way in NiFi to use a processor such as "PutFile" and have it write to one single file (append data to this file, or over-write the data in this file) - rather than create ...
- 6,270
9
votes
2
answers
2k
views
NIFI - Dev to Test to Prod
We are struggling with trying to figure out the best approach for updating processor configurations as a flow progresses through the dev, test, and prod stages. We would really like to avoid ...
- 9,121
9
votes
2
answers
10k
views
Apache Spark and Nifi Integration
I want to send Nifi flowfile to Spark and do some transformations in Spark and again send the result back to Nifi so that I can to further operations in Nifi. I don't want to write the flowfile ...
- 161
8
votes
2
answers
10k
views
Introduce time delay before moving flow files to next processor in NiFi
In NiFi, there exist a data flow to consume from MQTT (ConsumeMQTT) and publish into HDFS path (PutHDFS). I got a requirement to introduce 60 min delay before pushing the consumed data into HDFS path. ...
- 1,184
8
votes
3
answers
11k
views
Using flowfile size as an argument in RouteOnAttribute nifi
I want to know if is it possible to use flowfile size as an argument in routeOnAttribute i want to make expression like that :
${filename.fileSize>500}
but it tells me that ...
- 368
8
votes
2
answers
18k
views
In NiFi processor 'InvokeHTTP' where do you write body of POST request?
Before posting this question about Apache NiFi InvokeHTTP I have gone through all other questions and their answersbut I am still unsure the best flow I should have. My situation is as below:
1) From ...
- 279
8
votes
6
answers
8k
views
Post a NIFI template via REST?
I have multiple nifi servers that I would like to be able to POST templates to via the REST interface from a script
The "/controller/templates" endpoint appears to be the proper REST endpoint to ...
- 434
8
votes
2
answers
5k
views
NiFi from overwriting values in nifi.properties
I am running NiFi in docker with all relevant directories mounted as volumes. I am attempting to modify some settings in my nifi.properties file, specifically to add a custom properties file. However, ...
- 343
8
votes
1
answer
4k
views
How to stop the single processor in nifi1.1.0 using rest api
I am trying to stop the processor which is in running state using the PUT method in rest api /processors/{id}.
I am able to start the processor by changing the state in the component as follows "...
- 479
8
votes
1
answer
13k
views
NIFI: out of memory error inspite of increasing the heap size
I got myself into a problem where the NIFI service just refuse to start up. I monitored the nifi-app.log and found it was OutOfMemory error. I increased the memory in bootstap.conf file. But the error ...
- 394
8
votes
2
answers
7k
views
Apache NiFi - OutOfMemory Error: GC overhead limit exceeded on SplitText processor
I am trying to use NiFi to process large CSV files (potentially billions of records each) using HDF 1.2. I've implemented my flow, and everything is working fine for small files.
The problem is that ...
- 1,181
8
votes
2
answers
7k
views
Apache-Nifi Cannot create JDBC driver of class 'com.mysql.jdbc.Driver'
Output from $NIFI_HOME/logs/nifi-app.log:
2016-10-26 12:45:13,304 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.standard.PutSQL
org.apache.nifi.processor.exception.ProcessException: ...
8
votes
2
answers
3k
views
Spark concurrent writes on same HDFS location
I have a spark code which saves a dataframe to a HDFS location (date partitioned location) in Json format using append mode.
df.write.mode("append").format('json').save(hdfsPath)
sample hdfs location ...
- 105
7
votes
2
answers
8k
views
Apache Camel vs Apache Nifi
I am using Apache camel for quite long time and found it to be a fantastic solution for all kind of system integration related business need. But couple of years back I came accross the Apache Nifi ...
- 2,031
7
votes
3
answers
4k
views
Could not generate extensions' documentation when creating custom controller service in NIFI
I tried to create a custom controller service by using this guide. It is worked perfectly in nifi 1.9.2 version. But when I try with NIFI 1.11.3 version, I'm getting the following issue. I googled it, ...
- 2,767
7
votes
1
answer
4k
views
What is the purpose of data provenance in Apache NiFi Processors
For every processor there is a way to configure the processor and there is a context menu to view data provenance.
Is there a good explanation of what is data provenance?
- 2,940
7
votes
2
answers
8k
views
Apache Nifi - Extract Attributes From Avro
I'm trying to get my head around on extracting attributes from Avro and JSON. I'm able to extract attributes from JSON by using EvaluateJsonPath processor. I'm trying to do the same on Avro, but i'm ...
- 1,101
7
votes
1
answer
5k
views
EvaluateJsonPath unable to return a scalar
I'm trying to extract a value from JSON to a flowfile-attribute. When I run the EvaluateJsonPath processor I get an error stating
"Unable to get a scalar value for expression $..fields.storyBoard....
- 163
7
votes
1
answer
3k
views
Difference Between Processor Properties and Flowfile Attributes in Apache NiFi
My current understanding is that NiFi processor properties are specific to that processor. So adding a new property to a processor will only be visible within that processor and not be passed on to ...
- 4,700
7
votes
3
answers
9k
views
In NiFi is it possible to read selectively through FetchS3Object processor?
In Apache NiFi, using FetchS3Object to read from an S3 bucket, I see it can reads all the object in bucket and as they are added. Is it possible:
To configure the processor to read only objects added ...
- 161
7
votes
4
answers
5k
views
How to create new flow using Apache Nifi [closed]
How can I create new flows using Apache Nifi. Is there only one flow?
Also what about authentication? Can different users have different flows?
- 610
7
votes
2
answers
9k
views
Apache NiFi Unable to start the flow controller because the TLS configuration was invalid: The keystore properties are not valid
I am attempting to upgrade to Apache NiFi from 1.9.2 to 1.12.1 and no matter how I tweak the properties file, I keep getting errors about TLS. I was running just fine before the upgrade. I was able to ...
- 4,700
7
votes
3
answers
14k
views
NiFi - how to reference a flowFile in ExecuteStreamCommand?
I need to execute something like:
sed '1d' simple.tsv > noHeader.tsv
which will remove first line from my big flow file (> 1 GB).
The thing is - I need to execute it on my flow file, so it'd be:
...
- 1,490
7
votes
1
answer
5k
views
Difference between HDF and Apache NiFi
I am trying to understand difference between Apache Nifi and Hortonworks Data Flow (HDF).
How they differ from each other in terms of capability and overall design ? What will be possible use cases ...
- 2,704
7
votes
3
answers
4k
views
Does Apache NiFi support version control
I am trying to explore Apache NiFi. So far haven't seen any ways to version control flows.
Is there a way to version control flows when multiple users are trying to develop in the same instance?
...
- 71
7
votes
1
answer
11k
views
Split FlowContent by line and extract text to attributes NIFI
Im using NIFI and i want to extract attributes of my file lines .
My Filetext looks like this :
DEV=A9E ,SEN=1
DEV=B9E ,SEN=2
...
And i want to split text by line and then extract dev and sen to ...
- 1,267