Writing binary [] to ignite via memcache binary protocol

classic Classic list List threaded Threaded
10 messages Options
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Writing binary [] to ignite via memcache binary protocol

Hi, all,


New user to Ignite here. 

We are evaluating the possibility to replace memcached w/ Apache Ignite (v2.6). We notice recently that when we set a byte[] using net.spy.memcached client, and it seems sometimes the type of cache Object in Ignite is somehow set as String. Thus, when we get the value of the entry, it could not always convert back to binary[] properly, unless we specifically specified a byte[] transcoder when client performs get(key, transcoder) operation.

Does ignite internally have a way to store the data type when cache entry is stored? This would help a lot, since we might be working with driver of different languages as well. 

Thanks in advance!
Maxim.Pudov Maxim.Pudov
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, 

We have a libmemcached (C) client to write pure binary[]  to Ignite and another Java client to read from it.  We suspect Ignite stores it as String(UTF-8), perhaps related IGNITE-7028 - even from the packet dump, the data type is raw bytes (0)

Thanks and Regards,


On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

In reply to this post by Maxim.Pudov
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin
Павлухин Иван Павлухин Иван
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi Michael,

The troubles could be related to Python library. It seems in Python 2.7 there is no such thing as "byte array". And value passed to the client is string in this case.
I checked that Ignite recognizes bytes array type and stores in as byte array internally. I did following experiment with Spymemcached [1].
public class Memcached {
    public static void main(String[] args) throws IOException {
        MemcachedClient client = new MemcachedClient(
            new BinaryConnectionFactory(),
            AddrUtil.getAddresses("127.0.0.1:11211"));

        client.add("a", Integer.MAX_VALUE, new byte[]{1, 2, 3});
        client.add("b", Integer.MAX_VALUE, "123");

        System.out.println(Arrays.toString((byte[])client.get("a")));
        System.out.println(client.get("b"));

        System.exit(0);
    }
}

And I see expected output:
[1, 2, 3]
123


ср, 17 окт. 2018 г. в 10:25, Павлухин Иван <[hidden email]>:
Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin


--
Best regards,
Ivan Pavlukhin
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, 

Thanks for pointing out the mistake about the string data type. I used python3 which supports bytes type and run the test case. Ignite would seems still treat the received value as String type and would decode the data correctly. I would like to try with a some simple C client program and see how It works.

In addition, I checked the source code, and it seems the data type of key or value is 

Could someone share some insight of what  U.bytesToShort() does ?

Thanks.


On Wed, Oct 17, 2018 at 8:29 PM Павлухин Иван <[hidden email]> wrote:
Hi Michael,

The troubles could be related to Python library. It seems in Python 2.7 there is no such thing as "byte array". And value passed to the client is string in this case.
I checked that Ignite recognizes bytes array type and stores in as byte array internally. I did following experiment with Spymemcached [1].
public class Memcached {
    public static void main(String[] args) throws IOException {
        MemcachedClient client = new MemcachedClient(
            new BinaryConnectionFactory(),
            AddrUtil.getAddresses("127.0.0.1:11211"));

        client.add("a", Integer.MAX_VALUE, new byte[]{1, 2, 3});
        client.add("b", Integer.MAX_VALUE, "123");

        System.out.println(Arrays.toString((byte[])client.get("a")));
        System.out.println(client.get("b"));

        System.exit(0);
    }
}

And I see expected output:
[1, 2, 3]
123


ср, 17 окт. 2018 г. в 10:25, Павлухин Иван <[hidden email]>:
Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin


--
Best regards,
Ivan Pavlukhin
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hello!

It will just take two first bytes from extras and turn them into a numeric variable.

Later on, if flags & 0xff00 == 0x1000000b, then it is byte array, else it is string.

You should find a way to make your implementation send '-128' as fourth byte (If I understand endianness correctly). You could look it up in tcpdump.

Regards,

--
Ilya Kasnacheev


чт, 18 окт. 2018 г. в 10:33, Michael Fong <[hidden email]>:
Hi, 

Thanks for pointing out the mistake about the string data type. I used python3 which supports bytes type and run the test case. Ignite would seems still treat the received value as String type and would decode the data correctly. I would like to try with a some simple C client program and see how It works.

In addition, I checked the source code, and it seems the data type of key or value is 

Could someone share some insight of what  U.bytesToShort() does ?

Thanks.


On Wed, Oct 17, 2018 at 8:29 PM Павлухин Иван <[hidden email]> wrote:
Hi Michael,

The troubles could be related to Python library. It seems in Python 2.7 there is no such thing as "byte array". And value passed to the client is string in this case.
I checked that Ignite recognizes bytes array type and stores in as byte array internally. I did following experiment with Spymemcached [1].
public class Memcached {
    public static void main(String[] args) throws IOException {
        MemcachedClient client = new MemcachedClient(
            new BinaryConnectionFactory(),
            AddrUtil.getAddresses("127.0.0.1:11211"));

        client.add("a", Integer.MAX_VALUE, new byte[]{1, 2, 3});
        client.add("b", Integer.MAX_VALUE, "123");

        System.out.println(Arrays.toString((byte[])client.get("a")));
        System.out.println(client.get("b"));

        System.exit(0);
    }
}

And I see expected output:
[1, 2, 3]
123


ср, 17 окт. 2018 г. в 10:25, Павлухин Иван <[hidden email]>:
Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin


--
Best regards,
Ivan Pavlukhin
Michael Fong Michael Fong
Reply | Threaded
Open this post in threaded view
|

Re: Writing binary [] to ignite via memcache binary protocol

Hi, 

Thanks for pointing that out!

I checked the code where BYTE_ARR_FLAG = (8 << 8)
so I set flags = 2048 in my C client program - linking libmemcached library; so that Apache Ignite would recognize the SET value as byte array successfully. 
i.e.
rc = memcached_set(connection, key2, strlen(key2), value2, strlen(value2), (time_t)0, (uint32_t)2048);

Regards,


On Thu, Oct 18, 2018 at 6:12 PM Ilya Kasnacheev <[hidden email]> wrote:
Hello!

It will just take two first bytes from extras and turn them into a numeric variable.

Later on, if flags & 0xff00 == 0x1000000b, then it is byte array, else it is string.

You should find a way to make your implementation send '-128' as fourth byte (If I understand endianness correctly). You could look it up in tcpdump.

Regards,

--
Ilya Kasnacheev


чт, 18 окт. 2018 г. в 10:33, Michael Fong <[hidden email]>:
Hi, 

Thanks for pointing out the mistake about the string data type. I used python3 which supports bytes type and run the test case. Ignite would seems still treat the received value as String type and would decode the data correctly. I would like to try with a some simple C client program and see how It works.

In addition, I checked the source code, and it seems the data type of key or value is 

Could someone share some insight of what  U.bytesToShort() does ?

Thanks.


On Wed, Oct 17, 2018 at 8:29 PM Павлухин Иван <[hidden email]> wrote:
Hi Michael,

The troubles could be related to Python library. It seems in Python 2.7 there is no such thing as "byte array". And value passed to the client is string in this case.
I checked that Ignite recognizes bytes array type and stores in as byte array internally. I did following experiment with Spymemcached [1].
public class Memcached {
    public static void main(String[] args) throws IOException {
        MemcachedClient client = new MemcachedClient(
            new BinaryConnectionFactory(),
            AddrUtil.getAddresses("127.0.0.1:11211"));

        client.add("a", Integer.MAX_VALUE, new byte[]{1, 2, 3});
        client.add("b", Integer.MAX_VALUE, "123");

        System.out.println(Arrays.toString((byte[])client.get("a")));
        System.out.println(client.get("b"));

        System.exit(0);
    }
}

And I see expected output:
[1, 2, 3]
123


ср, 17 окт. 2018 г. в 10:25, Павлухин Иван <[hidden email]>:
Hi Michael,

Answering one of your questions.
> Does ignite internally have a way to store the data type when cache entry is stored?
Yes, internally Ignite maintains data types for stored keys and values.

Could you confirm that for real memcached your example works as expected? I will try reproduce your Python example. It should not be hard to check what exactly is stored inside Ignite.

ср, 17 окт. 2018 г. в 5:25, Michael Fong <[hidden email]>:
bump :)

Could anyone please help to answer a newbie question? Thanks in advance!

On Mon, Oct 15, 2018 at 4:22 PM Michael Fong <[hidden email]> wrote:
Hi, 

I kind of able to reproduce it with a small python script
import pylibmc

client = pylibmc.Client (["127.0.0.1:11211"], binary=True)


##abc
val = "abcd".decode("hex")
client.set("pyBin1", val)

print "val decode w/ iso-8859-1: %s" % val.encode("hex")

get_val = client.get("pyBin1")

print "Value for 'pyBin1': %s" % get_val.encode("hex")

where the the program intends to insert a byte[] into ignite using memcache binary protocol.
The output is 

val decode w/ iso-8859-1: abcd
Value for 'pyBin1': efbfbdefbfbd

where, 'ef bf bd' are the replacement character for UTF-8 String. Therefore, the value field seems to be treated as String in Ignite. 

Regards,

Michael



On Thu, Oct 4, 2018 at 9:38 PM Maxim.Pudov <[hidden email]> wrote:
Hi, it looks strange to me. Do you have a reproducer?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/


--
Best regards,
Ivan Pavlukhin


--
Best regards,
Ivan Pavlukhin