[Twisted-Python] txmongo vs. pymongo + callInThread

[Clayton Daley]

Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.

Clayton Daley

[Gelin Yan]

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton

Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.

Regards

gelin yan

[Clayton Daley]

Thanks. Are you aware of any resources that clarify which pymongo calls
are lazy and which actually result in calls to the Mongo server (thus
really benefit from derferToThread)?


Clayton Daley
***@gmail.com
513.505.1236

Post by Gelin Yan

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton
Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.
Regards
gelin yan
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[bret curtis]

Post by Gelin Yan

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton
Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.
Regards
gelin yan

Hey guys,

just wanted to drop in and say that txmongo isn't dead. We (Amplidata) just
recently sent a merge request to add SSL (authentication/encryption)
support[1] that Alexandre will review after the holidays. While Alexandre
isn't currently developing txmongo, he does review and accept pull
requests. So while the community is small, it is still alive.

Amplidata (the company I work for) has also expressed an interest in
getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and
its myriad forks to converge and development happening again.[2][3]

Is there any interest in seeing txmongo be developed further? What are some
of the pain points that people would rather defer to thread PyMongo instead
of using async txmongo?

Cheers,
Bret

[1] https://github.com/fiorix/mongo-async-python-driver/pull/67
[2] https://twistedmatrix.com/pipermail/twisted-python/2014-July/028489.html
[3] https://github.com/twisted/ldaptor

[Clayton Daley]

I'm still in "proof of concept" mode so take my comments with a grain of
salt... but my question was motivated by datetime issues:

- After switching to txmongo, timedelta seemed to break. Turns out it
breaks autobahn as well so I may need to solve the problem sooner... but
it's a material difference.
- I also wanted to take advantage of some timezone stuff purportedly
included in newer versions of pymongo (but I haven't fully implemented
anything yet to verify).

When it comes to interest in txmongo:

- Insofar as it would address the question of lazy calls (and possible
hassles like not-lazy iterators), it would be a welcome simplification to
deferToThread.
- At the same time, I don't see any intrinsic benefit to rebuild the
client all-the-way-down. If it were possible to wrap PyMongo in a
lazy-aware (and deferred returning) wrapper... that would be enough for me.
- If a wrapper reduces the work to maintain the project at parity, I'd
redouble my support for the approach. No need to commit volunteer
supporters (current and/or future) to more work than absolutely necessary.

Clayton Daley

Post by bret curtis

Post by Gelin Yan

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton
Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.
Regards
gelin yan

Hey guys,
just wanted to drop in and say that txmongo isn't dead. We (Amplidata)
just recently sent a merge request to add SSL (authentication/encryption)
support[1] that Alexandre will review after the holidays. While Alexandre
isn't currently developing txmongo, he does review and accept pull
requests. So while the community is small, it is still alive.
Amplidata (the company I work for) has also expressed an interest in
getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and
its myriad forks to converge and development happening again.[2][3]
Is there any interest in seeing txmongo be developed further? What are
some of the pain points that people would rather defer to thread PyMongo
instead of using async txmongo?
Cheers,
Bret
[1] https://github.com/fiorix/mongo-async-python-driver/pull/67
[2]
https://twistedmatrix.com/pipermail/twisted-python/2014-July/028489.html
[3] https://github.com/twisted/ldaptor
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Gelin Yan]

Post by bret curtis

Post by Gelin Yan

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton
Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.
Regards
gelin yan

Hey guys,
just wanted to drop in and say that txmongo isn't dead. We (Amplidata)
just recently sent a merge request to add SSL (authentication/encryption)
support[1] that Alexandre will review after the holidays. While Alexandre
isn't currently developing txmongo, he does review and accept pull
requests. So while the community is small, it is still alive.
Amplidata (the company I work for) has also expressed an interest in
getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and
its myriad forks to converge and development happening again.[2][3]
Is there any interest in seeing txmongo be developed further? What are
some of the pain points that people would rather defer to thread PyMongo
instead of using async txmongo?
Cheers,
Bret

Hi Bret

I am glad to know txmongo is still here. I used it several times a few
years ago and was impressed by its speed (twice faster than pymongo &
thread pool).

I hope txmongo can support the latest GEO index & replication.

Regards

gelin yan

[bret curtis]

Post by Gelin Yan

Post by bret curtis

Post by Gelin Yan

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread
instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
Clayton Daley

Hi Clayton
Txmongo has been inactive for a while so I suggest using pymongo &
twisted thread pool. I have been using them for a couple of years and they
work quite well.
Regards
gelin yan

Hey guys,
just wanted to drop in and say that txmongo isn't dead. We (Amplidata)
just recently sent a merge request to add SSL (authentication/encryption)
support[1] that Alexandre will review after the holidays. While Alexandre
isn't currently developing txmongo, he does review and accept pull
requests. So while the community is small, it is still alive.
Amplidata (the company I work for) has also expressed an interest in
getting txmongo up to speed with PyMongo, similar in how we got Ldaptor and
its myriad forks to converge and development happening again.[2][3]
Is there any interest in seeing txmongo be developed further? What are
some of the pain points that people would rather defer to thread PyMongo
instead of using async txmongo?
Cheers,
Bret

Hi Bret
I am glad to know txmongo is still here. I used it several times a few
years ago and was impressed by its speed (twice faster than pymongo &
thread pool).
I hope txmongo can support the latest GEO index & replication.
Regards
gelin yan

Hello Gelin,

You should check out our other thread about getting TxMongo moved into
Twisted's github org.
https://twistedmatrix.com/pipermail/twisted-python/2015-January/029144.html

There is already some support for Geo and replication, but it could always
use more work. What is it specifically you would like to see, aside from
everything. ;)

https://github.com/fiorix/txmongo/blob/master/txmongo/filter.py
^-- Support for GEO2D and GEOHAYSTACK spatial indices were added in 2012.

When you specify your hosts in ConnectionPool, you can give it a full
mongodb URI that includes all your replication servers, even with SSL
support. It is smart enough to cycle through until it finds a master to
continue operations.

I'm still trying to a compile a list of pain points that people have that
can be addressed. The more specific, the better.

Cheers,
Bret

[bret curtis]

We're not sure about what OP has done, but if there is ever anything
lacking in TxMongo... please open an issue/ticket with us over at:
https://github.com/twisted/txmongo

Cheers,
Bret

Post by Clayton Daley

Post by Clayton Daley
Are there any major disadvantages of using pymongo with callInThread

instead of txmongo? I'd like to take advantage of some newer features in
pymongo (unfortunately not available in txmongo) and it's certainly easier
to maintain feature parity using callInThread.
I know thread is a bit old, but I'm wondering if the OP ended up using
callInThread or not , and if there were any issues.
I started working on a txmongo solution to offload some storage, but then
realized I was already in a deferred thread (to handle blocking operations)
and i probably should be using pymongo.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Clayton Daley]

I (OP) was looking into these libraries/options as part of a side project
that's been delayed by real work. I'm also relegating Mongo to a caching
layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB"
and realizing I was going to have the exact same problems.

Clayton Daley

Post by bret curtis
We're not sure about what OP has done, but if there is ever anything
https://github.com/twisted/txmongo
Thanks for the reply! I actually did open a few lowball tickets on
TxMongo yesterday -- the package doesn't specify the minimum required
versions for mongo or twisted.
I did some tests yesterday on PyMongo, and it seems like using it in
deferToThread is safe. I also ran some simulations on staging under a
heavy load; no exceptions, everything passes, data is stored correctly.
The current pymongo "client" appears to be a threadsafe connection pool,
and I can just grab a connection from it when needed in a thread. Woo Hoo!
If the next version of this tool stays in Python, we'll definitely need to
use TxMongo. In the current design, we're already deferred to a thread and
would have to restructure too much to use TxMongo.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[bret curtis]

There is one thing that TxMongo does that PyMongo, Motor and the rest don't
do: handle deadlines and timeouts.

We're (HGST/WesternDigital) using MongoDB (with WiredTiger) in real-time
applications and one of our requirements was to honour application set
deadlines (or timeouts) on a per-call basis.

So if you are interested in that, we have it. :)

For those just tuning in...
Timeout: how many seconds the call is allowed run before raising an error.
Deadline: a time (unix timestamp) in the future that the call is allowed to
run before raising an error.

We take advantage of Twisted's native deferred cancellation mechanism in
addition to validation checks that no modifications happen to MongoDB when
a TimeExceeded is raised.

We've been very satisfied with how MongoDB has performed (post 3.0/WT)
along with TxMongo and Twisted.

Cheers,
Bret

Post by Clayton Daley
I (OP) was looking into these libraries/options as part of a side project
that's been delayed by real work. I'm also relegating Mongo to a caching
layer after stumbling upon Sarah Mei's "Why You Should Never Use MongoDB"
and realizing I was going to have the exact same problems
Thanks for the update!
1. We use redis for KV caching, and its really, really, really fast.
2. Riak can use LevelDB as a backing store -- leveldb supports snappy
compression on documents
3. If you need the document search, i really like Elastic
I don't really like/trust mongodb from past experience myself. I would
normally not use it for anything other than a read-through cache... except
for my current need. We have a web spider and need to archive the
snapshots. WireTiger, which was bought by mongodb and integrated into the
3.x branch, is the only storage backend I've found that will do "bucket
level compression". I forget if it's applied on pages or blocks, but it
will basically apply zlib or snappy compression onto sets of documents. I
looked for weeks on something that could do that, it was the only option.
We did a lot of storage tests on our typical data, and bucketed compression
saves way more space than document compression or document compression
using a seed.
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Gelin Yan]

Post by bret curtis
There is one thing that TxMongo does that PyMongo, Motor and the rest
don't do: handle deadlines and timeouts.
We're (HGST/WesternDigital) using MongoDB (with WiredTiger) in real-time
applications and one of our requirements was to honour application set
deadlines (or timeouts) on a per-call basis.
So if you are interested in that, we have it. :)
For those just tuning in...
Timeout: how many seconds the call is allowed run before raising an error.
Deadline: a time (unix timestamp) in the future that the call is allowed
to run before raising an error.
We take advantage of Twisted's native deferred cancellation mechanism in
addition to validation checks that no modifications happen to MongoDB when
a TimeExceeded is raised.
We've been very satisfied with how MongoDB has performed (post 3.0/WT)
along with TxMongo and Twisted.
Cheers,
Bret

I have switched from pymongo to txmongo for about two months. I got at
least 2x faster on query. Despite of some minor issues which have been
fixed recently, I am very happy with it.

Regards

gelin yan

[Glyph Lefkowitz]

I have switched from pymongo to txmongo for about two months. I got at least 2x faster on query. Despite of some minor issues which have been fixed recently, I am very happy with it.

This is really cool to hear. I try not to say "twisted will make everything faster" because speed is a complex question and a component like Twisted is rarely ever the one answer you need... but it's so nice when it just does :-D.

-glyph