Writing binary [] to ignite via memcache binary protocol

classic Classic list List threaded Threaded
7 messages Options
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Writing binary [] to ignite via memcache binary protocol

Hi, all,


New user to Ignite here. 

We are evaluating the possibility to replace memcached w/ Apache Ignite (v2.6). We notice recently that when we set a byte[] using net.spy.memcached client, and it seems sometimes the type of cache Object in Ignite is somehow set as String. Thus, when we get the value of the entry, it could not always convert back to binary[] properly, unless we specifically specified a byte[] transcoder when client performs get(key, transcoder) operation.

Does ignite internally have a way to store the data type when cache entry is stored? This would help a lot, since we might be working with driver of different languages as well. 

Thanks in advance!
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, 

We have a libmemcached (C) client to write pure binary[]  to Ignite and another Java client to read from it.  We suspect Ignite stores it as String(UTF-8), perhaps related IGNITE-7028 - even from the packet dump, the data type is raw bytes (0)

Thanks and Regards,


On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

In reply to this post by Maxim.Pudov
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi Michael,

The troubles could be related to Python library. It seems in Python 2.7 there is no such thing as "byte array". And value passed to the client is string in this case.
I checked that Ignite recognizes bytes array type and stores in as byte array internally. I did following experiment with Spymemcached [1].
public class Memcached {
    public static void main(String[] args) throws IOException {
        MemcachedClient client = new MemcachedClient(
            new BinaryConnectionFactory(),
            AddrUtil.getAddresses("127.0.0.1:11211"));

        client.add("a", Integer.MAX_VALUE, new byte[]{1, 2, 3});
        client.add("b", Integer.MAX_VALUE, "123");

        System.out.println(Arrays.toString((byte[])client.get("a")));
        System.out.println(client.get("b"));

        System.exit(0);
    }
}

And I see expected output:
[1, 2, 3]
123


ср, 17 окт. 2018 г. в 10:25, Павлухин Иван <[hidden email]>:
Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin


--
Best regards,
Ivan Pavlukhin