BindaryObject exception failing to read classes and serialzaiton issues

classic Classic list List threaded Threaded
5 messages Options
Pascoe11 Pascoe11
Reply | Threaded
Open this post in threaded view
|

BindaryObject exception failing to read classes and serialzaiton issues

This post was updated on .
Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I
made a mistake so I will provide my question here. Excuse me if I am being
impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the
server node configuration. I will attache the output of the stack trace at
the bottom, however I first wish to explain what the software does and how
we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a
TcpDiscoveryVmIpFinder and dont set anything other than a port range for
the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts
a service and this in turn starts a grpc server, I have placed some scala
code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above
mentioned computeTask as a variable, I am not sure if this could be the
cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc
server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc
stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems.
However, adding more nodes and trying to execute the new tasks on a remote
node or on a node that has a certain attribute gives me a massive stack
trace in the face.
Basically, if I want to execute a task on a node where the service and grpc
server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder
of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field
[name=server]
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at
org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
deserialize object [typeName=io.grpc.internal.ServerImpl]
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
read field [name=registry]
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to
deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Question on submitted post

Hi,

It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect.
What you should do is
- not pass gRPC to the service instance
- add an init() method implementation to your service
- in your init() start your gRPC server

Stan

On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <[hidden email]> wrote:
Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I made a mistake so I will provide my question here. Excuse me if I am being impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the server node configuration. I will attache the output of the stack trace at the bottom, however I first wish to explain what the software does and how we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts a service and this in turn starts a grpc server, I have placed some scala code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above mentioned computeTask as a variable, I am not sure if this could be the cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems. However, adding more nodes and trying to execute the new tasks on a remote node or on a node that has a certain attribute gives me a massive stack trace in the face.
Basically, if I want to execute a task on a node where the service and grpc server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=server]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.ServerImpl]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=registry]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe
Pascoe11 Pascoe11
Reply | Threaded
Open this post in threaded view
|

Re: Question on submitted post

Hi Stan,

Thanks for your response. I have tried this, but it has not fixed the issue.
The grpc server class was moved into the service where the interface methods "init","execute" and "cancel" perform an initialization of the serve, as well start and stopping respectively. But this was already implemented.

Does this issue not have something to do with class loading? The second node that is started doesn't have any information of the custom classes for serialization. I have also moved all JARS of these custom classes into the libs folder in ignite but that has also not helped.

On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <[hidden email]> wrote:
Hi,

It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect.
What you should do is
- not pass gRPC to the service instance
- add an init() method implementation to your service
- in your init() start your gRPC server

Stan

On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <[hidden email]> wrote:
Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I made a mistake so I will provide my question here. Excuse me if I am being impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the server node configuration. I will attache the output of the stack trace at the bottom, however I first wish to explain what the software does and how we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts a service and this in turn starts a grpc server, I have placed some scala code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above mentioned computeTask as a variable, I am not sure if this could be the cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems. However, adding more nodes and trying to execute the new tasks on a remote node or on a node that has a certain attribute gives me a massive stack trace in the face.
Basically, if I want to execute a task on a node where the service and grpc server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=server]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.ServerImpl]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=registry]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe
Stanislav Lukyanov Stanislav Lukyanov
Reply | Threaded
Open this post in threaded view
|

Re: Question on submitted post

The message "Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]" means InternalHandlerRegistry is being sent between the nodes - which it shouldn't be.
What you need to do is to find where is it being sent. You shouldn't pass any gRPC to any Ignite configuration, or, I believe, to any Ignite method.
Also, be careful with compute jobs that use lambdas and anonymous classes. By their nature they catch objects around them too easy, so often you'll see a compute job accidentally bringing some context over the network. The best way to avoid it is to use `public static` classes instead of anonymous classes and lambdas - verbose but bulletproof.

If you still have an issue with this please share more complete code and config example. A small reproducer on github would be ideal.

Stan

On Mon, Aug 26, 2019 at 4:09 PM Pascoe Scholle <[hidden email]> wrote:
Hi Stan,

Thanks for your response. I have tried this, but it has not fixed the issue.
The grpc server class was moved into the service where the interface methods "init","execute" and "cancel" perform an initialization of the serve, as well start and stopping respectively. But this was already implemented.

Does this issue not have something to do with class loading? The second node that is started doesn't have any information of the custom classes for serialization. I have also moved all JARS of these custom classes into the libs folder in ignite but that has also not helped.

On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <[hidden email]> wrote:
Hi,

It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect.
What you should do is
- not pass gRPC to the service instance
- add an init() method implementation to your service
- in your init() start your gRPC server

Stan

On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <[hidden email]> wrote:
Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I made a mistake so I will provide my question here. Excuse me if I am being impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the server node configuration. I will attache the output of the stack trace at the bottom, however I first wish to explain what the software does and how we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts a service and this in turn starts a grpc server, I have placed some scala code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above mentioned computeTask as a variable, I am not sure if this could be the cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems. However, adding more nodes and trying to execute the new tasks on a remote node or on a node that has a certain attribute gives me a massive stack trace in the face.
Basically, if I want to execute a task on a node where the service and grpc server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=server]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.ServerImpl]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=registry]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe
Pascoe11 Pascoe11
Reply | Threaded
Open this post in threaded view
|

Re: Question on submitted post

Hi Stan,

Thanks for the info! Indeed this was a very big mistake and your explanation made it clear what the error was.

I was passing an instance of the grpc server to the compute job, a part of the code which I had neglected to add in the description above.

Cheers!

On Mon, Aug 26, 2019 at 7:21 PM Stanislav Lukyanov <[hidden email]> wrote:
The message "Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]" means InternalHandlerRegistry is being sent between the nodes - which it shouldn't be.
What you need to do is to find where is it being sent. You shouldn't pass any gRPC to any Ignite configuration, or, I believe, to any Ignite method.
Also, be careful with compute jobs that use lambdas and anonymous classes. By their nature they catch objects around them too easy, so often you'll see a compute job accidentally bringing some context over the network. The best way to avoid it is to use `public static` classes instead of anonymous classes and lambdas - verbose but bulletproof.

If you still have an issue with this please share more complete code and config example. A small reproducer on github would be ideal.

Stan

On Mon, Aug 26, 2019 at 4:09 PM Pascoe Scholle <[hidden email]> wrote:
Hi Stan,

Thanks for your response. I have tried this, but it has not fixed the issue.
The grpc server class was moved into the service where the interface methods "init","execute" and "cancel" perform an initialization of the serve, as well start and stopping respectively. But this was already implemented.

Does this issue not have something to do with class loading? The second node that is started doesn't have any information of the custom classes for serialization. I have also moved all JARS of these custom classes into the libs folder in ignite but that has also not helped.

On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov <[hidden email]> wrote:
Hi,

It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect.
What you should do is
- not pass gRPC to the service instance
- add an init() method implementation to your service
- in your init() start your gRPC server

Stan

On Thu, Aug 22, 2019 at 10:52 AM Pascoe Scholle <[hidden email]> wrote:
Hi there,

How long does it usually take, for a post to be seen on the forum? Maybe I made a mistake so I will provide my question here. Excuse me if I am being impatient:


=========================================
Good afternoon everyone,

I have recently run into an issue and I think the problem lies in the server node configuration. I will attache the output of the stack trace at the bottom, however I first wish to explain what the software does and how we are using ignite.

I start multiple server nodes with peerClassEnabled set to true, using  a TcpDiscoveryVmIpFinder and dont set anything other than a port range for the ipFinder.

Using the REST protocol a ComputeTaskAdapter task is executed which starts a service and this in turn starts a grpc server, I have placed some scala code to show what I mean.

class StartService extends ComputeTaskAdapter[String, Any]{
  var ignite: Ignite = null;
  @IgniteInstanceResource
  def setIgnite(someIgnite: Ignite): Unit = {
    ignite = someIgnite
  }

 def map(...)={
...
// port is an integer
val server = new GrpcServer(ignite, port);

val service = new ServiceImpl(name, server);
/*
within the method execute of the Service interface, server.start() is called
*/

val serviceconfig = new ServiceConfiguration();
          serviceconfig.setName(name);
          serviceconfig.setTotalCount(1);
          serviceconfig.setMaxPerNodeCount(1);
          ignite.services().deploy(serviceconfig);
...
}

}

this task returns a map with some non important variables.

The grpc server takes the ignite instance created within the above mentioned computeTask as a variable, I am not sure if this could be the cause of the issue.

Using grpc protocol, we create a ComputeTask which is executed by the grpc server some more code below:

class GrpcServer(val ignite:Ignite, val port:Int) extends ..Some Grpc stuff..{

def someGrpcProtocol(request: Message):Future[String]={
val newTask = new SomeTask();

ignite.compute(ignite.cluster()).execute(newTask, someinput);
Future("Request is being processed");
}

}


If a single server node is started, the program runs without problems. However, adding more nodes and trying to execute the new tasks on a remote node or on a node that has a certain attribute gives me a massive stack trace in the face.
Basically, if I want to execute a task on a node where the service and grpc server do not reside, the exception happens.

I have placed all custom classes within a jar that lies in the libs folder of the ignite-bin project.
We are currently on version 2.7

If you require anything else just let me know, ill be on it asap.

Thanks for any help that may come my way.

Cheers!

Here is most of the stack trace:
class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=server]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:313)
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:102)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10140)
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:440)
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1119)
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1923)
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.ServerImpl]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 29 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=registry]
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
        ... 34 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=io.grpc.internal.InternalHandlerRegistry]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:914)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1764)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1984)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:703)
        at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:188)
        ... 35 more

=======================================

Thanks and kind regards,
Pascoe