deepcopy() is extremely slow
PythonOptimizationDeep CopyPython Problem Overview
I have a game state in Python with about 1000 objects (planetary systems + stars + planets), and I need to copy it and apply a bunch of transformations to it when requested. However, at about 1 request/second, this is taking up 24.63% of my runtime. How can I make it go fast? Note that copying less is not an option, since the transforms touch just about everything.
EDIT: got it down to 8% with judicious implementation of __deepcopy__ on things. Still, not good enough. (Good enough is 1% or less, I plan on throwing many more things at this.) timeit says 41.8ms per deepcopy().
Python Solutions
Solution 1 - Python
Actually, deepcopy is very slow. But we can use json, ujson, or cPickle. we can use json/cPickle to dump an object, and load it later. This is my test:
Total time: 3.46068 s
File: test_deepcopy.py
Function: test at line 15
Line # Hits Time Per Hit % Time Line Contents
==============================================================
15 @profile
16 def test():
17 100 957585 9575.9 27.7 b = deepcopy(a)
18 100 862 8.6 0.0 c = copy(a)
19 100 42295 422.9 1.2 d = ujson.loads(ujson.dumps(a))
20 100 85040 850.4 2.5 e = json.loads(json.dumps(a))
21 100 2323465 23234.7 67.1 f = pickle.loads(pickle.dumps(a, -1))
22 100 51434 514.3 1.5 g = cPickle.loads(cPickle.dumps(a, -1))
as what we can see, json/ujson/cPickle is faster than deepcopy, but pickle...
Solution 2 - Python
If you create your own class to hold these objects you can create your own methods that work with copy and deep copy. http://www.rafekettler.com/magicmethods.html#copying</s> (Broken Link)
New Link for a github repository https://github.com/RafeKettler/magicmethods
class MyClass():
def __copy__(self):
copy_object = MyClass()
return copy_object
def __deepcopy__(self, memodict={}):
copy_object = MyClass()
copy_object.value = self.value
return copy_object
if __name__ == "__main__":
my_inst = MyClass()
print(copy.deepcopy(my_inst))
Here is a similar description from the previous broken link.
Copying
Sometimes, particularly when dealing with mutable objects, you want to be able to copy an object and make changes without affecting what you copied from. This is where Python's copy comes into play. However (fortunately), Python modules are not sentient, so we don't have to worry about a Linux-based robot uprising, but we do have to tell Python how to efficiently copy things.
__copy__(self)
Defines behavior for copy.copy() for instances of your class. copy.copy() returns a shallow copy of your object -- this means that, while the instance itself is a new instance, all of its data is referenced -- i.e., the object itself is copied, but its data is still referenced (and hence changes to data in a shallow copy may cause changes in the original).
__deepcopy__(self, memodict={})
Defines behavior for copy.deepcopy() for instances of your class. copy.deepcopy() returns a deep copy of your object -- the object and its data are both copied. memodict is a cache of previously copied objects -- this optimizes copying and prevents infinite recursion when copying recursive data structures. When you want to deep copy an individual attribute, call copy.deepcopy() on that attribute with memodict as the first argument. What are some use cases for these magic methods? As always, in any case where you need more fine-grained control than what the default behavior gives you. For instance, if you are attempting to copy an object that stores a cache as a dictionary (which might be large), it might not make sense to copy the cache as well -- if the cache can be shared in memory between instances, then it should be.
Solution 3 - Python
I've made a fast experiment comparing both deepcopy/json/ujson for several cases and my results contradicts @cherish's ones on certain cases, posting the little experiment here:
import ujson
import timeit
import json
import random
import string
import copy
import ujson
import sys
def random_string(N):
return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))
def random_json(width=5, height=5, levels=1):
dct = {}
lst = [random_string(4) for i in range(width)]
lst2 = [random.randint(0, 10000) for i in range(width)]
lst3 = [bool(random.randint(0, 1)) for i in range(width)]
for j in range(height):
dct[str(j)] = lst
dct[str(width+j)] = lst2
dct[str(2*width+j)] = lst3
for i in range(levels):
new_dct = {}
for j in range(height):
new_dct[str(j)] = dct
dct = json.loads(json.dumps(new_dct))
return new_dct
if __name__ == "__main__":
print(sys.version)
levels = 3
for i in range(15):
dataset = random_json(i, i, levels)
print("Comparing deepcopy/ujson/json using random dataset({},{},{}), length {}".format(i,i,levels, len(json.dumps(dataset))))
print(timeit.timeit('copy.deepcopy(dataset)',
setup='from __main__ import copy, dataset', number=10))
print(timeit.timeit('ujson.loads(ujson.dumps(dataset))',
setup='from __main__ import ujson, dataset', number=10))
print(timeit.timeit('json.loads(json.dumps(dataset))',
setup='from __main__ import json, dataset', number=10))
print()
And the results would be:
3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
Comparing deepcopy/ujson/json using random dataset(0,0,3), length 2
2.6842977659931844e-05
0.00012039864979822371
7.776568527950847e-05
Comparing deepcopy/ujson/json using random dataset(1,1,3), length 63
0.0002731667726569534
3.552747043226263e-05
0.00012987264191349377
Comparing deepcopy/ujson/json using random dataset(2,2,3), length 1106
0.0011858280130946362
0.00034974820892205325
0.0007093651596308467
Comparing deepcopy/ujson/json using random dataset(3,3,3), length 6834
0.0042218477363672215
0.0021178319874343293
0.003378267688436718
Comparing deepcopy/ujson/json using random dataset(4,4,3), length 26572
0.011379054029782284
0.006288757016181971
0.009920059244030693
Comparing deepcopy/ujson/json using random dataset(5,5,3), length 79210
0.028879491215043435
0.027906433274870912
0.029595961868760734
Comparing deepcopy/ujson/json using random dataset(6,6,3), length 183678
0.047142979515255284
0.04682125853300759
0.06791747047568517
Comparing deepcopy/ujson/json using random dataset(7,7,3), length 395528
0.08239215142913198
0.09871347134571351
0.15347433002098887
Comparing deepcopy/ujson/json using random dataset(8,8,3), length 764920
0.1351954464835896
0.19448842613700734
0.3020533693660834
Comparing deepcopy/ujson/json using random dataset(9,9,3), length 1356570
0.24560258734724671
0.44074906118659407
0.5705849913806413
Comparing deepcopy/ujson/json using random dataset(10,10,3), length 2287770
0.3237815755327835
0.61104051671153
0.8698565598118777
Comparing deepcopy/ujson/json using random dataset(11,11,3), length 3598750
0.4958284828467452
0.9472223636741877
1.2514314609961668
Comparing deepcopy/ujson/json using random dataset(12,12,3), length 5636414
0.6261448233909714
1.4066722957969802
1.8636325417418167
Comparing deepcopy/ujson/json using random dataset(13,13,3), length 8220800
0.8396582099444547
2.061675688670409
2.755659427352441
Comparing deepcopy/ujson/json using random dataset(14,14,3), length 12018290
1.0951926990258762
2.96703050743886
4.088875914783021
Conclusion from this little experiment is:
- When dictionaries are small ones
time(ujson)<time(json)<time(deepcopy) - When dictionaries are big ones
time(deepcopy)<time(ujson)<time(json)
So it depends the number of copies you're making per second and which type of dictionary you're dealing with, you'll prefer switching between deepcopy or ujson.
Solution 4 - Python
base on @BPL's test program and add marshal on my ARMv6-compatible processor
print(timeit.timeit('marshal.loads(marshal.dumps(dataset))',
setup='from __main__ import marshal, dataset', number=1))
marshal is fast than ujson and support set and tuple
2.7.14 (default, Mar 6 2019, 13:27:55)
[GCC 7.3.0]
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,1), length 2
0.000588178634644
0.000134944915771
0.000258922576904
0.00113606452942
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,3), length 2
0.000546932220459
0.000134944915771
0.000180006027222
0.00120401382446
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,0,5), length 2
0.000545978546143
0.000128984451294
0.000185966491699
0.00106000900269
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,1), length 50
0.00154900550842
0.000281810760498
0.000414848327637
0.00174903869629
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,3), length 242
0.00655102729797
0.000789880752563
0.00133085250854
0.00432300567627
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,2,5), length 1010
0.0514280796051
0.0015549659729
0.00413513183594
0.0148711204529
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,1), length 172
0.00250005722046
0.000365018844604
0.000761985778809
0.00263404846191
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,3), length 2892
0.0329101085663
0.00363397598267
0.0110101699829
0.0262169837952
()
Comparing deepcopy/marshal/ujson/json using random dataset(0,4,5), length 46412
0.616458892822
0.0826110839844
0.189103841782
0.504135131836
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,1), length 2
0.000693082809448
0.000132083892822
0.000182867050171
0.00107002258301
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,3), length 2
0.000566005706787
0.000132083892822
0.000180959701538
0.00107598304749
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,0,5), length 2
0.000562906265259
0.000128984451294
0.000184059143066
0.00118517875671
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,1), length 258
0.00405406951904
0.000534057617188
0.00124287605286
0.00309610366821
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,3), length 1058
0.026270866394
0.00180387496948
0.00363302230835
0.0096640586853
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,2,5), length 4338
0.0778729915619
0.00682806968689
0.0151469707489
0.0468928813934
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,1), length 716
0.00720596313477
0.00100684165955
0.0215280056
0.0062358379364
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,3), length 11468
0.112984895706
0.0238728523254
0.0448131561279
0.0874760150909
()
Comparing deepcopy/marshal/ujson/json using random dataset(2,4,5), length 183628
1.83552503586
0.407335042953
0.617804050446
1.65498495102
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,1), length 2
0.000571012496948
0.000132083892822
0.000189781188965
0.00121593475342
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,3), length 2
0.000757932662964
0.000131130218506
0.000180959701538
0.00144195556641
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,0,5), length 2
0.00056791305542
0.000132083892822
0.000184059143066
0.00107407569885
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,1), length 430
0.00451302528381
0.00053596496582
0.00142502784729
0.00343203544617
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,3), length 1730
0.0259549617767
0.00232696533203
0.00387692451477
0.0187470912933
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,2,5), length 7026
0.112207174301
0.0119769573212
0.0211799144745
0.0547370910645
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,1), length 1684
0.00609397888184
0.00121903419495
0.00452899932861
0.00959086418152
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,3), length 26828
0.19367814064
0.0293428897858
0.0688338279724
0.140627145767
()
Comparing deepcopy/marshal/ujson/json using random dataset(4,4,5), length 433484
3.54843020439
0.590909004211
1.09412097931
2.72070598602
Solution 5 - Python
You can provide your own copy functions to the objects such that you won't need deep copy. deep copy inspects every object to check what needs to be copied. This is an expensive operation.