Wednesday, 1st October 2008
I recently ventured into some strange waters, namely Google App Engine, to create a tiny geolocation app to scratch an itch of mine which I didn’t want to burden my slice with. But pretty early I stumbled upon the limitations of App Engine.
My project uses a massive database to map the IP’s to their country of origin which proved to be quite difficult since App Engine doesn’t allow multiple inequality filters which turns out to be a huge pain.
Rather than this simple GQL call:
# Fetch all rows
result = db.GqlQuery('SELECT * FROM Country WHERE ipTo >= %s AND ipFrom <= %s ' % (numeric_ip,numeric_ip) ).get()
Which would fetch the only row that I’m looking for, I had to use this workaround:
# Fetch all rows
rows = db.GqlQuery('SELECT * FROM Country WHERE ipTo >= %s' % numeric_ip ).fetch(1000)
# Find the correct row
result = []
for entity in rows:
if entity.ipFrom <= numeric_ip:
result.append(entity)
Which both puts a bit more stress on the server and is just plain ugly, but at least it works.
There is also no way to easily empty a table or start over, which is really annoying when you are importing 85,000 rows and some of the transfers are cancelled halfway which creates some serious database bloat.
App Engine is a great platform but there are some flaws, but I doubt I would use it for anything bigger than small apps in the future. If you’re able to overlook & overcome the various annoyances I would highly recommend to at least take App Engine out for a spin.
DusK
The reason App Engine doesn’t support this type of query is because optimizing multiple-inequality queries is hard - and it’s an uncommon enough case that optimizing for a maximum of one inequality is much easier to implement and runs much faster.
Ian Clarke
DusK, your comment is a non-sequitur. The fact that doing it would be hard in no way lessons the impact of not having the feature.
Hsiu-Fan Wang
I think you may want to change your data model a bit to fit the “performs fast” sweet spot of the datastore.
One thing that you could also do is use a filter so that the code is a bit prettier.
Ian Clarke: it doesn’t lessen the impact of not having a feature, but by creating APIs that limit you to performant operations its easier to write code that’s well… performant. I think that was kind of the design goal of App Engine, take everything Google has learned about building scalable websites and give people a toolkit to do so. Whether all sites need to scale though is a different issue.
Andreas Schipplock
Ok, so what about the flaws…you mentioned there are several but you only mentioned this database thingy…
Ludwig Pettersson
Andreas, these are the flaws that I encountered - both concerning the datastore, not every flaw there is.
But I guess it’s a pretty good trade off for getting to use Googles infrastructure that can scale like there’s no tomorrow.
Michael
“…you can’t start over which creates some serious database bloat.”
You can start over, but it does require a little work. You need to fetch and delete all entities, then vaccuum the index and update.
Ludwig Pettersson
Michael, it was worded quite badly - what I meant was there is no easy way to empty a table or start over.
Of course there are ways, I explored some scripts that would delete 1,000 rows at a time and just loop 85 times - but it’s not straight forward and sure requires some work.
Michael
Sorry, I just wanted to point that out in case anyone reading might not know the procedure.
I really would like to see Google add some more administrative tools to their control panel, most especially import/export CSV functions (with support for compressed files to reduce transfer time).
Tony Arkles
I’ve been working full-time on an App Engine based app, and it definitely requires a change in thinking.
The App Engine datastore does definitely require a bit of a change in thinking; it’s not SQL, and things that make sense in SQL don’t necessarily make sense for fast scalable data access.
Would you be able to post a sample chunk of the data you’re trying to query? I’ve got an idea in my head about how to structure the data for efficient datastore lookups, but without actually seeing the data, it’s hard to say.
Ludwig Pettersson
Michael, additions to the control panel would be great - having to use scripts to import CSV files is not really a pleasant experience. Would have made my life a lot easier.
Tony, sure thing. This is a tiny copy & paste bit from the Data Viewer:
Tony Arkles
Apologies for the delayed response…
I’m quite surprised actually… this seems like a much trickier problem than it looks at first glance :)
Based on my understanding of how the app engine indexing happens, I think that checking rows[-1] first might speed it up a bit (you don’t need to iterate and check each row), but that doesn’t fix the problem that there’d still be a huge datastore fetch.
… there doesn’t look to me to be an easy solution to this though.
Bromley
Is it possible that you could get it to work efficiently with a query like:
ipFrom >= yourTestIp ORDER BY ipFrom
It looks to me like ordering the results by ipFrom would you enable to get the right one on the first fetch (of just one entity). When you’ve fetched the entity you can compare yourTestIp with ipTo: if yourTestIp <= ipTo you’ve got the country, otherwise there’s nothing stored for that IP.
Quite likely I’m missing something obvious but that’s looking like a good solution to me at the moment…
SilldicaSailm
miley cyrus nude http://wetpixel.com/forums/index.php?showuser=28152 - miley cyrus nude miley cyrus nude http://forums.buddytv.com/members/nerokilser.html - miley cyrus nude miley cyrus nude http://blogcastrepository.com/members/booraeder.aspx - miley cyrus nude kim kardashian nude http://wetpixel.com/forums/index.php?showuser=28156 - kim kardashian nude
LiSO_Jul
Dear Author www.luddep.se ! Choice at you hard
abibiot
TENNESSEE CASH ADVANCE middot TEXAS CASH ADVANCE middot UTAH CASH ADVANCE This is a. Approve method of reimbursement of overnight travel status.
Vay mua nha
Great post, thank you.
Djubrail
djubi test http://twitter.com/legalhighsherb legal high
Djubrail
djubi test http://twitter.com/legalhighsherb legal high
moreWeile
good news health care antivirale oregon state mineral http://usadrugstoretoday.com/products/plan-b.htm novelty tea pots http://usadrugstoretoday.com/categories/general-health.htm mineral bath spa hotels arkansas speman hypertension recipe hand job penis
freefrazonnow
Udjieower udjieower http://udjieower.com udjieower http://udjieower.com
wainnardy
porn child fashion lolita http://topcitystyle.com/celyn-b—classic-denim-brand123.html bridesmaids shoes hot rod fashion mail order catalog for shoes http://topcitystyle.com/32-new-size13.html gucci purse nude male clothes female
wawavargo
how to build snow shoes http://topcitystyle.com/bacci-abbracci-brand57.html filipina fashion model portfolio argentina interior designers robert land shoes http://topcitystyle.com/orange-men-color121.html designer scrubs problems faced by graphic designers before the use of multimedia software
theolobislosy
how to get whiteout out of clothes http://topcitystyle.com/xl-gucci-size6.html lauren active interior designer sample resume chocolate designer packaging http://topcitystyle.com/bikkembergs-t-shirt-with-chained-metal-logo-for—item726.html amber rayne melissa lauren video european comfort shoes
wawavargo
personal designer used clothing shopper http://topcitystyle.com/white-armani-color4.html french teen fashion designs pvc shoes recent ralph lauren bedding collections in red http://topcitystyle.com/on-sale-page32.html chanel cake eyeliner video game designer jobs
wainnardy
lauren michelle hill gallery http://topcitystyle.com/women-apos-s-t-shirts-armani-category62.html wide lady shoes jeff kleinman thom mccan shoes http://topcitystyle.com/skirts-category96.html chacu shoes uk keen clearance shoes
Royanysfulson
unusual botique fashions http://topcitystyle.com/40-dress-shirts-size27.html old fashion pot roast diesel shoes women ladies in fetish clothes http://topcitystyle.com/leopard-tunic-color176.html lauren hill gallery chinese fashion
Coryseeseeata
golf clothes women http://topcitystyle.com/black-silver-shoes-color111.html preventing crepe soled shoes from squeaking ralph laurenpickstitch stovepipe simple shoes http://topcitystyle.com/shoes-gucci-category12.html guga kuerten clothes mens italian oxford shoes
Wanynilia
jim st laurent scout http://topcitystyle.com/black-green-red-casual-color76.html shoes industry directory of india munroe shoes designer sectional furniture http://topcitystyle.com/blue-men-color32.html gothic cloing and shoes ecko james shoes
Wanynilia
chanel myspace layout http://topcitystyle.com/l-tank-tops-size7.html fashion dress finn comfort shoes dian ferrari shoes australia http://topcitystyle.com/-top-and-skirt-roberto-cavalli-category106.html womans designer clothing samples fashion design schools
DotreargoGone
charles albert shoes http://topcitystyle.com/casual-page5.html snow shoes for dag walking magazine adds and polo and ralph lauren giada de laurentiis nude http://topcitystyle.com/52-jeans-size3.html peart izumi spinning shoes long island pond and waterfall designers
Clummatar
clothes steamer http://topcitystyle.com/dolce-amp-gabbana-tank-top-white-black-item1907.html timberland shower in clothes kelly shoes interior designers presentation boards http://topcitystyle.com/killah-dressy-black-shirt-item1119.html career brand name shoes sales schutzkleinspannung
DotreargoGone
designer handbags http://topcitystyle.com/guess-t-shirts-for-men-brand21.html calvin klein sunglasses lauren london desk top screen savers china clothes holesell http://topcitystyle.com/xs-women-s-tops-size46.html casual shoes zara womens clothes
taitlybreerty
chaps ralph lauren http://topcitystyle.com/smiley-world-brand103.html jodan team flow shoes chanel logo earrings naked girls taking off each others clothes http://topcitystyle.com/black-leggings-pants-type-color2.html modeling and fashion and summer camp and virginia designer takes flight with stuffed birds
DiteDyday
matthew hellyerretro lamp designers http://topcitystyle.com/energie-unisex-messenger-bag-beige-item1407.html chanel myspace layout walmart maid clothes naot shoes http://topcitystyle.com/new-casual-tops-type6.html tuti fruti baby clothes henschenklein
Lophealppiema
chanel pearls http://topcitystyle.com/richmond-luxury-t-shirt-for-men-white-item1337.html wide ladies shoes dog walking shoes design your own fashion http://topcitystyle.com/brown-beige-accessories-color183.html fashion buyer salaries coco chanel designs
nishhineHof
cato fashion store http://topcitystyle.com/roberto-cavalli-long-sleeve-top-for-women-purple-item2508.html magazine adds and polo and ralph lauren montgomery county comcast chanel lineup fashion malls in maryland http://topcitystyle.com/?action=products&product_id=2227 lauren joseph searfoss designer portraits
DiteDyday
pics of chanel perfume http://topcitystyle.com/?action=products&product_id=2074 discount medical shoes crocs fashion sketching wading shoes http://topcitystyle.com/men copies of fashion stylist resume kenmore clothes dryer metal shudder type idler pulley
Lophealppiema
discount designer prom dresses http://topcitystyle.com/cream-women-s-tops-color68.html designers fabric organic baby clothes lightning bolt clothes thrown off why http://topcitystyle.com/blumarine-tank-top-for-women-white-item2018.html puma ferrari shoes ladies comfort shoes
nishhineHof
sexy girls in stockings and shoes http://topcitystyle.com/navy-blue-brown-dress-shirts-color214.html native mexican fashion discovery chanel judas icariote infant white lace shoes http://topcitystyle.com/black-white-dresses-color23.html chanel cuff bracelet family album fashion catalogue
ownsaed
http://needman.ru знакомства с иностранцами знакомства с иностранцами
legeBessaxy
basketball flight nike shoes http://luxefashion.us/tommy-hilfiger-men-brand97.html russian ballet shoes yahama brake shoes westminster fashion http://luxefashion.us/purple-accessories-color37.html lacoste shoes designer wallets
legeBessaxy
paul thomas shoes http://luxefashion.us/multi-color-roberto-cavalli-color180.html timber designers fashions for mother of the bride toddler tap shoes http://luxefashion.us/white-grey-red-blue-leather-shoes-color223.html laurence harbor jeff kleinman
LatwoodoCleld
saint peter chanel georgia http://luxefashion.us/blue-on-sale-color32.html tall womens clothes designer dress top wedding free label designer http://luxefashion.us/-club-wear-category69.html horse shoes fashion knitting
utessulmoma
jessica simpson shoes paris http://www.thefashionhouse.us/yellow-tank-tops-color44.html running shoes invented a machine in which shoes are firmly attached to soles gymboree clothes lines http://www.thefashionhouse.us/navy-blue-casual-color21.html womens wide width shoes hp palm pilot designer accessories
Dillon Branch
Very great post. Honest.
opigonoavaibe
gucci sneakers http://www.thefashionhouse.us/white-pink-blue-yellow-men-color107.html best shoes for plantar fashiitis finn comfort shoes clothes steamer http://www.thefashionhouse.us/?action=products&product_id=2046 ballet shoes best ralph lauren active wear
AftendVef
birch run outlet shoes http://luxefashion.us/dolce-amp-gabbana-athletic-shorts-for-men-black-item2572.html recent ralph lauren bedding collections in red chic career clothes orthotic shoes http://luxefashion.us/jackets-amp-sweatshirts-dolce-amp-gabbana-category10.html belks womens shoes gucci furniture
opigonoavaibe
fashion stylists located in chicago il http://www.thefashionhouse.us/classic-denim-page4.html tubbs odyssey snowshoes clothes homemade extra wide shoes http://www.thefashionhouse.us/leather-shoes-page6.html fashion buyer salaries dollhouse shoes
WP Themes
Amiable dispatch and this mail helped me alot in my college assignement. Thank you for your information.
Wordpress Themes
Nice post and this mail helped me alot in my college assignement. Thanks you as your information.
porno
thanks for share nice comment
rottweiler
thanks for all information to give us.