How to speed up this code? On average, work takes about 4 minutes for a group of 300 people. I think the thing is in the entry of groups in the dictionary. Is there any more rational way to do this?

This code takes all the subscribers of a particular group (in this case, fineline_official), then checks with each subscriber what groups it consists of groups = api.users.getSubscriptions(user_id = mid)['groups'] . Then it adds the id of these groups to the dictionary. If the dictionary does not contain an id of any group, then it adds it there groups_dict[gid] = 1 . If the id of this group is already in the dictionary, then it adds 1 to the key value. At the entrance we get a group (id), at the output of the 10 most popular groups among the subscribers of this group

 import vk session = vk.Session(access_token='token') #VK token api = vk.API(session) group_id = 'fineline_official' #group id group_stat=api.groups.getMembers(group_id = group_id ) #ΠŸΠΎΠ»ΡƒΡ‡Π°Π΅ΠΌ участников Π³Ρ€ΡƒΠΏΠΏΡ‹ members_id = group_stat['users'] members_count = group_stat['count'] groups_dict = {} if members_count <= 1000: for mid in members_id: try: groups = api.users.getSubscriptions(user_id = mid)['groups'] groups_id = groups['items'] for gid in groups_id: try: groups_dict[gid] groups_dict[gid] += 1 except: groups_dict[gid] = 1 except: continue else: print('>1000') lst = [] for k in groups_dict.items(): #Π”ΠΎΠ±Π°Π²Π»Π΅Π½ΠΈΠ΅ Π² список ΠΊΠΎΡ€Ρ‚Π΅ΠΆ ΠΈΠ· id Π³Ρ€ΡƒΠΏΠΏΡ‹ ΠΈ ΠΊΠΎΠ»-Π²Π° участников lst.append(k) lst_sort = sorted(lst, key=lambda x: x[1]) #Π‘ΠΎΡ€Ρ‚ΠΈΡ€ΠΎΠ²ΠΊΠ° ΠΏΠΎ ΠΊΠΎΠ»-Π²Ρƒ участников print (lst_sort) lst1 = lst_sort[-10:] #10 самых популярных print(lst1)
  • Make sure there are fewer calls to the server. Generally use the tools at hand. Yes, and ask questions would be nice to learn. - 0andriy
  • @ 0andriy I did not know how to formulate a question more correctly. About the means at hand did not quite understand. Do you mean parsing? - r4d1f
  • To find a bottleneck. You have not even done your homework ... I expressed the version, but I do not have confirmed data. You and cards in hand, as soon as you find a bottleneck, then you can substantively talk. Suddenly you have a 9600 baud modem? - 0andriy
  • 2
    You need not to think, but to measure and look for a bottleneck - andreymal
  • one
    By the way, the code can be simplified (regardless of speed): top10groups = collections.Counter(g.id for m in members(group) for g in groups(m.id)).most_common(10) . Try another connection pool (requests module) and group requests (more than one subscriber at a time β€” 25 at a time if you believe the answer below). - jfs

2 answers 2

Excerpt from VK documentation:

  • Limitations and Recommendations
    • 3.1. Frequency restrictions VKontakte API methods (except for the methods from the secure and ads sections) can be accessed no more than 3 times per second. If the logic of your application involves calling several methods in a row, it makes sense to pay attention to the execute method. It allows you to make up to 25 calls to different methods in a single request.
  • It is not that. In my code, the most frequent request occurs in the groups = api.users.getSubscriptions(user_id = mid)['groups'] . But in order to reach it again, you need to perform a subsequent cycle, which exactly takes longer than 0.33 seconds - r4d1f
  • @ r4d1f you measured at least how much time this line is executed? - andreymal

With execute, you can implement it like this:

 import vk import math token ="" session = vk.Session(access_token=token) api = vk.API(session) group_id = "fineline_official" members = api.groups.getMembers(group_id=group_id, v="5.62") # сколько ΠΏΡ€ΠΎΡ…ΠΎΠ΄ΠΎΠ² потрСбуСтся, Ссли Π±Ρ€Π°Ρ‚ΡŒ ΠΈΠ· списка ΠΏΠΎ 25 элСмСнтов # Π΄Π΅Π»ΠΈΠΌ ΠΊΠΎΠ»-Π²ΠΎ элСмнтов Π½Π° 25 ΠΈ округляСм Π² Π±ΠΎΠ»ΡŒΡˆΡƒΡŽ сторону num_of_pass = math.ceil(members["count"] / 25) groups_dict = dict() for npass in range(num_of_pass): pass_users = list() # Π·Π°Π±ΠΈΡ€Π°Π΅ΠΌ ΠΈΠ· списка 25 элСмСнтов for i in range(npass * 25, npass * 25 + 25): if i < members["count"]: pass_users.append(str(members["items"][i])) # ΠΊΠΎΠ΄, ΠΊΠΎΡ‚ΠΎΡ€Ρ‹ΠΉ ΠΌΡ‹ Π±ΡƒΠ΄Π΅ΠΌ ΠΏΠ΅Ρ€Π΅Π΄Π°Π²Π°Ρ‚ΡŒ code = """var u = [""" + ','.join(pass_users) + """]; var i = 0; var r = []; while(i < u.length) { r = r + API.groups.get({"user_id":u[i]}); i = i + 1; } return r;""" users_groups = api.execute(code = code); print(users_groups) for gid in users_groups: try: groups_dict[gid] += 1 except: groups_dict[gid] = 1