Comment web gratter les adeptes de Instagram navigateur web?

MODIFIER: Décembre 2018 mise à Jour:

De choses ont changé dans Insta terre depuis que cela a été posté. Voici un script de mise à jour qui est un peu plus pythonic et une meilleure utilisation de XPATH/CSS chemins.

Notez que pour utiliser ce script de mise à jour, vous devez installer le explicit paquet (pip install explicit), ou de convertir chaque ligne avec waiter à un pur sélénium explicite attendre.

import itertools
from explicit import waiter, XPATH
from selenium import webdriver
def login(driver):
username = ""  # <username here>
password = ""  # <password here>
# Load page
driver.get("https://www.instagram.com/accounts/login/")
# Login
waiter.find_write(driver, "//div/input[@name='username']", username, by=XPATH)
waiter.find_write(driver, "//div/input[@name='password']", password, by=XPATH)
waiter.find_element(driver, "//div/button[@type='submit']", by=XPATH).click()
# Wait for the user dashboard page to load
waiter.find_element(driver, "//a/span[@aria-label='Find People']", by=XPATH)
def scrape_followers(driver, account):
# Load account page
driver.get("https://www.instagram.com/{0}/".format(account))
# Click the 'Follower(s)' link
# driver.find_element_by_partial_link_text("follower").click()
waiter.find_element(driver, "//a[@href='/instagram/followers/']", by=XPATH).click()
# Wait for the followers modal to load
waiter.find_element(driver, "//div[@role='dialog']", by=XPATH)
# At this point a Followers modal pops open. If you immediately scroll to the bottom,
# you hit a stopping point and a "See All Suggestions" link. If you fiddle with the
# model by scrolling up and down, you can force it to load additional followers for
# that person.
# Now the modal will begin loading followers every time you scroll to the bottom.
# Keep scrolling in a loop until you've hit the desired number of followers.
# In this instance, I'm using a generator to return followers one-by-one
follower_css = "ul div li:nth-child({}) a.notranslate"  # Taking advange of CSS's nth-child functionality
for group in itertools.count(start=1, step=12):
for follower_index in range(group, group + 12):
yield waiter.find_element(driver, follower_css.format(follower_index)).text
# Instagram loads followers 12 at a time. Find the last follower element
# and scroll it into view, forcing instagram to load another 12
# Even though we just found this elem in the previous for loop, there can
# potentially be large amount of time between that call and this one,
# and the element might have gone stale. Lets just re-acquire it to avoid
# that
last_follower = waiter.find_element(driver, follower_css.format(follower_index))
driver.execute_script("arguments[0].scrollIntoView();", last_follower)
if __name__ == "__main__":
account = 'instagram'
driver = webdriver.Chrome()
try:
login(driver)
# Print the first 75 followers for the "instagram" account
print('Followers of the "{}" account'.format(account))
for count, follower in enumerate(scrape_followers(driver, account=account), 1):
print("\t{:>3}: {}".format(count, follower))
if count >= 75:
break
finally:
driver.quit()

J'ai fait un rapide test pour montrer comment le rendement diminue de façon exponentielle le plus de followers que vous essayez de gratter de cette façon:

$ python example.py
Followers of the "instagram" account
Found    100 followers in 11 seconds
Found    200 followers in 19 seconds
Found    300 followers in 29 seconds
Found    400 followers in 47 seconds
Found    500 followers in 71 seconds
Found    600 followers in 106 seconds
Found    700 followers in 157 seconds
Found    800 followers in 213 seconds
Found    900 followers in 284 seconds
Found   1000 followers in 375 seconds

Post Original:
Votre question est un peu confuse. Par exemple, je ne suis pas vraiment sûr de ce que "à partir de laquelle je peux gratter et de la pagination à travers toutes les itérations" signifie réellement. De quoi êtes-vous en train d'utiliser pour gratter et paginer?

Peu importe, instagram.com/instagram/media/ n'est pas le même type de point de terminaison comme instagram.com/instagram/followers. Le media point de terminaison semble être une API REST, configuré pour retourner facilement un parseable objet JSON.

La followers point de terminaison n'est pas vraiment une bonne terminaison de ce que je peux dire. Plutôt, Instagram AJAXs dans les informations à la source de la page (à l'aide de Réagir?) après vous cliquez sur les Disciples bouton. Je ne pense pas que vous serez en mesure d'obtenir ces informations sans l'aide de quelque chose comme le Sélénium, qui permet de charger/rendre le javascript qui affiche les adeptes de l'utilisateur.

Cet exemple de code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def login(driver):
username = ""  # <username here>
password = ""  # <password here>
# Load page
driver.get("https://www.instagram.com/accounts/login/")
# Login
driver.find_element_by_xpath("//div/input[@name='username']").send_keys(username)
driver.find_element_by_xpath("//div/input[@name='password']").send_keys(password)
driver.find_element_by_xpath("//span/button").click()
# Wait for the login page to load
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.LINK_TEXT, "See All")))
def scrape_followers(driver, account):
# Load account page
driver.get("https://www.instagram.com/{0}/".format(account))
# Click the 'Follower(s)' link
driver.find_element_by_partial_link_text("follower").click()
# Wait for the followers modal to load
xpath = "//div[@style='position: relative; z-index: 1;']/div/div[2]/div/div[1]"
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, xpath)))
# You'll need to figure out some scrolling magic here. Something that can
# scroll to the bottom of the followers modal, and know when its reached
# the bottom. This is pretty impractical for people with a lot of followers
# Finally, scrape the followers
xpath = "//div[@style='position: relative; z-index: 1;']//ul/li/div/div/div/div/a"
followers_elems = driver.find_elements_by_xpath(xpath)
return [e.text for e in followers_elems]
if __name__ == "__main__":
driver = webdriver.Chrome()
try:
login(driver)
followers = scrape_followers(driver, "instagram")
print(followers)
finally:
driver.quit()

Cette approche est problématique pour un certain nombre de raisons, la principale d'entre elles étant de savoir comment ralentir il est, par rapport à la fonction de l'API.

OriginalL'auteur Levi Noecker