Write python dataframe to ignite table.

classic Classic list List threaded Threaded
7 messages Options
favas favas
Reply | Threaded
Open this post in threaded view
|

Write python dataframe to ignite table.

Hi,

 

Is there a way to bulk load python dataframe values to ignite table?

 

Regards,

Favas 

 

dmagda dmagda
Reply | Threaded
Open this post in threaded view
|

Re: Write python dataframe to ignite table.

Andrey, Alexey, Nikolay,

Do we have any solutions for calling Ignite + Spark implementation from Python?

-
Denis


On Tue, Oct 22, 2019 at 3:41 AM Muhammed Favas <[hidden email]> wrote:

Hi,

 

Is there a way to bulk load python dataframe values to ignite table?

 

Regards,

Favas 

 

stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Write python dataframe to ignite table.

In reply to this post by favas
What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:


Regards,
Stephen

On 22 Oct 2019, at 11:41, Muhammed Favas <[hidden email]> wrote:

Hi,
 
Is there a way to bulk load python dataframe values to ignite table?
 
Regards,
Favas 


favas favas
Reply | Threaded
Open this post in threaded view
|

RE: Write python dataframe to ignite table.

Hi,

 

I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.

Now I want to bulk insert the whole dataframe into ignite table without looping through.

 

The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.

 

Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.

 

 

Regards,

Favas 

 

From: Stephen Darlington <[hidden email]>
Sent: Monday, October 28, 2019 5:05 PM
To: [hidden email]
Subject: Re: Write python dataframe to ignite table.

 

What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:

 

 

Regards,

Stephen



On 22 Oct 2019, at 11:41, Muhammed Favas <[hidden email]> wrote:

 

Hi,

 

Is there a way to bulk load python dataframe values to ignite table?

 

Regards,

Favas 

 

stephendarlington stephendarlington
Reply | Threaded
Open this post in threaded view
|

Re: Write python dataframe to ignite table.

Ah, sorry. When I see “DataFrame” I immediately think of Spark.

While that would be a great addition, I don’t think the Python thin-client currently supports DataFrames. I think your two options are to use Spark DataFrames or loop through, using put/putAll to import that data. Three options if you include creating a patch for the thin-client ;)

Regards,
Stephen

On 29 Oct 2019, at 10:04, Muhammed Favas <[hidden email]> wrote:

Hi,
 
I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.
Now I want to bulk insert the whole dataframe into ignite table without looping through.
 
The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.
 
Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.
 
 
Regards,
Favas 
 
From: Stephen Darlington <[hidden email]> 
Sent: Monday, October 28, 2019 5:05 PM
To: [hidden email]
Subject: Re: Write python dataframe to ignite table.
 
What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:
 
 
Regards,
Stephen


On 22 Oct 2019, at 11:41, Muhammed Favas <[hidden email]> wrote:
 
Hi,
 
Is there a way to bulk load python dataframe values to ignite table?
 
Regards,
Favas 


ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Write python dataframe to ignite table.

In reply to this post by favas
Hello!

Well, COPY command does allow you to do column mapping:

COPY FROM '/path/to/local/file.csv'
INTO tablename (columnName, columnName, ...) FORMAT CSV

If you need to do non-trivial transformations, you can use JDBC driver in SET STREAMING ON mode.

Regards,
--
Ilya Kasnacheev


вт, 29 окт. 2019 г. в 13:11, Muhammed Favas <[hidden email]>:

Hi,

 

I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.

Now I want to bulk insert the whole dataframe into ignite table without looping through.

 

The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.

 

Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.

 

 

Regards,

Favas 

 

From: Stephen Darlington <[hidden email]>
Sent: Monday, October 28, 2019 5:05 PM
To: [hidden email]
Subject: Re: Write python dataframe to ignite table.

 

What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:

 

 

Regards,

Stephen



On 22 Oct 2019, at 11:41, Muhammed Favas <[hidden email]> wrote:

 

Hi,

 

Is there a way to bulk load python dataframe values to ignite table?

 

Regards,

Favas 

 

favas favas
Reply | Threaded
Open this post in threaded view
|

RE: Write python dataframe to ignite table.

Hi Ilya,

 

Thanks for the response!!

 

My requirement to have the flexibility to map the csv column to specific table column.

The file I may be getting will have more columns and I want to import some specific set of columns to my table. I assume that flexibility is not available currently in COPY command.

 

Right now I have achieved it through spark dataframe for mapping the column and bulk insert into ignite. It is moreover giving nearly same performance as ignite COPY command.

 

Regards,

Favas 

 

From: Ilya Kasnacheev <[hidden email]>
Sent: Tuesday, October 29, 2019 8:46 PM
To: [hidden email]
Subject: Re: Write python dataframe to ignite table.

 

Hello!

 

Well, COPY command does allow you to do column mapping:

 

COPY FROM '/path/to/local/file.csv'
INTO tablename (columnName, columnName, ...) FORMAT CSV

 

If you need to do non-trivial transformations, you can use JDBC driver in SET STREAMING ON mode.

 

Regards,

--

Ilya Kasnacheev

 

 

вт, 29 окт. 2019 г. в 13:11, Muhammed Favas <[hidden email]>:

Hi,

 

I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.

Now I want to bulk insert the whole dataframe into ignite table without looping through.

 

The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.

 

Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.

 

 

Regards,

Favas 

 

From: Stephen Darlington <[hidden email]>
Sent: Monday, October 28, 2019 5:05 PM
To: [hidden email]
Subject: Re: Write python dataframe to ignite table.

 

What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:

 

 

Regards,

Stephen

 

On 22 Oct 2019, at 11:41, Muhammed Favas <[hidden email]> wrote:

 

Hi,

 

Is there a way to bulk load python dataframe values to ignite table?

 

Regards,

Favas