Store raw binary value in Apache Ignite through thin python client

classic Classic list List threaded Threaded
4 messages Options
kulinskyvs kulinskyvs
Reply | Threaded
Open this post in threaded view
|

Store raw binary value in Apache Ignite through thin python client

This post was updated on .
I'm trying to save some raw binary data into Apache Ignite using thin Python client and the process is very slow (actually it looks like it depends on the size of the data). For my simple test case I've started locally Ignite with single node (version 2.7.0 without any specific configuration):
bin/ignite.sh 
Also, I have a binary file with the content I want to store in Ignite. The size if around 6MB. Here is my source code:
with open("/var/bin_text.bin", mode='rb') as file:
    file_content = file.read()

client = Client()
client.connect('127.0.0.1', 10800)
my_cache = client.get_or_create_cache('my cache')
my_cache.put("key_bin", file_content)
After my_cache.put the process just freezes. I haven't even managed to wait till the end (I've terminated it after 2 minutes). However, passing the file content transformed into a string works very fast
my_cache.put("key_str", str(file_content))
What I'm missing? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Denis Mekhanikov Denis Mekhanikov
Reply | Threaded
Open this post in threaded view
|

Re: Store raw binary value in Apache Ignite through thin python client

Hi!

Thanks for the report! Seems like the implementation of serialization of primitive arrays is not optimal.
I created a JIRA ticket for this issue: https://issues.apache.org/jira/browse/IGNITE-11854

Denis

вт, 14 мая 2019 г. в 13:53, kulinskyvs <[hidden email]>:
I'm trying to save some raw binary data into Apache Ignite using thin Python
client and the process is very slow (actually it looks like it depends on
the size of the data).

For my simple test case I've started locally Ignite with single node
(version 2.7.0 without any specific configuration):


Also, I have a binary file with the content I want to store in Ignite. The
size if around 6MB. Here is my source code:


After *my_cache.put *the process just freezes. I haven't even managed to
wait till the end (I've terminated it after 2 minutes).

However, passing the file content transformed into a string works very fast
.


What I'm missing?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
kulinskyvs kulinskyvs
Reply | Threaded
Open this post in threaded view
|

Re: Store raw binary value in Apache Ignite through thin python client

Hi,

Thank you for your feedback!
Just for information - I've also created a stackoverflow question
(https://stackoverflow.com/questions/56127565/store-raw-binary-value-in-apache-ignite-through-thin-python-client)
related to this topic > I'll post the JIRA issue link there.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
YevIgn YevIgn
Reply | Threaded
Open this post in threaded view
|

Re: Store raw binary value in Apache Ignite through thin python client

Hi.

I think the issue is related to building bytes counterpart of an array during serialization using += operation on bytes object which is immutable. This leads to throwing away previously created bytes instances, each of which grows at new iteration.

This aligns with what I observe when run the test provided here.

Quick fix - https://github.com/YevIgn/ignite/commit/64211b2d7c9e174f0c8979cbb4f26df52b8ebcd8

After it, the array building stops hanging on my machine.

Should I open PR?

Note: First saw the issue, the message here is copy from my comment in it.

Best regards,
Evgeniy Ignatiev.

On 5/16/2019 4:42 PM, kulinskyvs wrote:
Hi,

Thank you for your feedback!
Just for information - I've also created a stackoverflow question
(https://stackoverflow.com/questions/56127565/store-raw-binary-value-in-apache-ignite-through-thin-python-client)
related to this topic > I'll post the JIRA issue link there.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/