Unintended merging of multiple transactions into a single transaction

classic Classic list List threaded Threaded
2 messages Options
kimec.ethome.sk kimec.ethome.sk
Reply | Threaded
Open this post in threaded view
|

Unintended merging of multiple transactions into a single transaction

Greetings,

while testing our Apache Ignite based app with Gatling, I have noticed
that sometimes a write to a single cache fails, because it has different
writeBehind setting than some other cache. The problem is that the two
caches are not part of the same transaction and should have been updated
in two different transactions, not one.

I do not have a reproducer at hand but the scenario is simple:
* Have server node bundled with your application
* Have at least two nodes
* Have two caches with PARTITIONED and TRANSACTIONAL
* Have one of the caches with writeBehind=true and the other one with
writeBehind=false
* Use optimistic transaction with the first cache (SERIALIZED +
OPTIMISTIC)
* Update both caches in sequence with two separate transactions but
demarcate the first transaction with commitAsync() instead of commit()
* Have non-Ignite fixed size thread pool initiating the commits
* Shoot your application down with Gatling with no pauses or delays to
stress the whole system

After some time you may get "Failed to enlist new cache to existing
transaction" errors for operations that should not have been committed
as a single transaction since the transaction boundary was demarcated
with commitAsync().

I've tried to narrow down the issue to these sections:

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxStateImpl.java#L266-L271

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTransactionalCacheAdapter.java#L1112

My guess is that IgniteTxStateImpl is somehow thread local/or cached and
under heavy load in a fixed sized thread pool, separate transactions are
merged into one "batch" for optimization. The trouble is that
commitAsync() in this case does not demarcate your transactions as you
would have guessed.

Is this the expected behavior?

--
S pozdravom,

Kamil Mišúth
ilya.kasnacheev ilya.kasnacheev
Reply | Threaded
Open this post in threaded view
|

Re: Unintended merging of multiple transactions into a single transaction

Hello!

I think that commitAsync() is not enough, you need to close() transaction explicitly before starting a new one. Just a hunch.

Regards,
--
Ilya Kasnacheev


ср, 5 июн. 2019 г. в 10:19, kimec.ethome.sk <[hidden email]>:
Greetings,

while testing our Apache Ignite based app with Gatling, I have noticed
that sometimes a write to a single cache fails, because it has different
writeBehind setting than some other cache. The problem is that the two
caches are not part of the same transaction and should have been updated
in two different transactions, not one.

I do not have a reproducer at hand but the scenario is simple:
* Have server node bundled with your application
* Have at least two nodes
* Have two caches with PARTITIONED and TRANSACTIONAL
* Have one of the caches with writeBehind=true and the other one with
writeBehind=false
* Use optimistic transaction with the first cache (SERIALIZED +
OPTIMISTIC)
* Update both caches in sequence with two separate transactions but
demarcate the first transaction with commitAsync() instead of commit()
* Have non-Ignite fixed size thread pool initiating the commits
* Shoot your application down with Gatling with no pauses or delays to
stress the whole system

After some time you may get "Failed to enlist new cache to existing
transaction" errors for operations that should not have been committed
as a single transaction since the transaction boundary was demarcated
with commitAsync().

I've tried to narrow down the issue to these sections:

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxStateImpl.java#L266-L271

https://github.com/apache/ignite/blob/07addc2660feceabf0fe7190ba9d8c83be80d4e9/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTransactionalCacheAdapter.java#L1112

My guess is that IgniteTxStateImpl is somehow thread local/or cached and
under heavy load in a fixed sized thread pool, separate transactions are
merged into one "batch" for optimization. The trouble is that
commitAsync() in this case does not demarcate your transactions as you
would have guessed.

Is this the expected behavior?

--
S pozdravom,

Kamil Mišúth