Flink Streamer

classic Classic list List threaded Threaded
3 messages Options
kadsank kadsank
Reply | Threaded
Open this post in threaded view
|

Flink Streamer

This post has NOT been accepted by the mailing list yet.
Hi, I am working on real time streaming data analysis using Apache Flink-1.1.3. My system consist of Kafka cluster for message queue, Flink cluster which read the messages from kafka partitions and make some analysis on it and finally I want to dump the generated data into Ignite Cache. For the system I am using IgniteSink class to sink the data into ignite cache. The versions are as follows: Flink 1.1.3 Kafka 2.10 Ignite 2.0.0 When I tried to run the job on flink cluster it gives me the following error, `Exception in thread "main" org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:409) at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:95) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:382) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:374) at org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.executeRemotely(RemoteStreamEnvironment.java:209) at org.apache.flink.streaming.api.environment.RemoteStreamEnvironment.execute(RemoteStreamEnvironment.java:173) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1429) at flink_ignite_sink_remote.main(flink_ignite_sink_remote.java:77) Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$mcV$sp(JobManager.scala:822) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.sink.flink.IgniteSink$SinkContext$Holder at org.apache.ignite.sink.flink.IgniteSink$SinkContext.getStreamer(IgniteSink.java:201) at org.apache.ignite.sink.flink.IgniteSink$SinkContext.access$100(IgniteSink.java:175) at org.apache.ignite.sink.flink.IgniteSink.invoke(IgniteSink.java:165) at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:373) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:358) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:346) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:329) at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51) at flink_ignite_sink_remote$Splitter.flatMap(flink_ignite_sink_remote.java:177) at flink_ignite_sink_remote$Splitter.flatMap(flink_ignite_sink_remote.java:1) at org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:48) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:373) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:358) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:346) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:329) at org.apache.flink.streaming.api.operators.StreamSource$NonTimestampContext.collect(StreamSource.java:161) at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecord(AbstractFetcher.java:225) at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.run(Kafka09Fetcher.java:253) at java.lang.Thread.run(Thread.java:745)` As I have bold the text of error, I have included all the libraries of Ignite-Flink into my project, still I got this error. Can you please help me out? благодаря(Thanks).
dkarachentsev dkarachentsev
Reply | Threaded
Open this post in threaded view
|

Re: Flink Streamer

Hi,

I'm not quite familiar with Flink and Kafka connectors, but I'm going to figure them out and come back to help you.

Thanks!

-Dmitry.
Roman Shtykh Roman Shtykh
Reply | Threaded
Open this post in threaded view
|

Re: Flink Streamer

Hi,

Can you check your logs and see if "Ignition.start(igniteCfgFile)" line of the Holder [1] starts the grid without failures?

-Roman



On Thursday, February 16, 2017 7:16 PM, dkarachentsev <[hidden email]> wrote:


Hi,

I'm not quite familiar with Flink and Kafka connectors, but I'm going to
figure them out and come back to help you.

Thanks!

-Dmitry.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Flink-Streamer-tp10650p10663.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.