[Solved] Using Selenium with Python and PhantomJS to download file to filesystem

I’ve been grappling with using PhantomJS/Selenium/python-selenium to download a file to the filesystem.
I’m able to easily navigate through the DOM and click, hover etc. Downloading a file is, however, proving to be quite troublesome. I’ve tried a headless approach with Firefox and pyvirtualdisplay but that wasn’t working well either and was unbelievably slow. I know That CasperJS allows for file downloads. Does anyone know how to integrate CasperJS with Python or how to utilize PhantomJS to download files. Much appreciated.

Solution #1:

Despite this question is quite old, downloading files through PhantomJS is still a problem. But we can use PhantomJS to get download link and fetch all needed cookies such as csrf tokens and so on. And then we can use requests to download it actually:

import requests
from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('page_with_download_link')
download_link = driver.find_element_by_id('download_link')
session = requests.Session()
cookies = driver.get_cookies()

for cookie in cookies: 
    session.cookies.set(cookie['name'], cookie['value'])
response = session.get(download_link)

And now in response.content actual file content should appear. We can next write it with open or do whatever we want.

Respondent: valignatev

Solution #2:

PhantomJS doesn’t currently support file downloads. Relevant issues with workarounds:

As far as I understand, you have at least 3 options:

  • switch to casperjs (and you should leave python here)
  • try with headless on xvfb
  • switch to normal non-headless browsers

Here are also some links that might help too:

Respondent: alecxe

Solution #3:

My use case required a form submission to retrieve the file. I was able to accomplish this using the driver’s execute_async_script() function.

 js = '''
    var callback = arguments[0];
    var theForm = document.forms['theFormId'];
    data = new FormData();
    data.append('eventTarget', "''' + target + '''"); // this is the id of the file clicked
    data.append('otherFormField', theForm.otherFormField.value);

    var xhr = new XMLHttpRequest();
    xhr.open('POST', theForm.action, true);
'''

for cookie in driver.get_cookies():
    js += ' xhr.setRequestHeader("' + cookie['name'] + '", "' + cookie['value'] + '"); '

js += '''
    xhr.onload = function () {
        callback(this.responseText);
    };
    xhr.send(data);
'''

driver.set_script_timeout(30)
file = driver.execute_async_script(js)
Respondent: dnbwise

Solution #4:

Is not posible in that way. You can use other alternatives to download files like wget o curl.

Use firefox to find the right request and selenium to get the values for that and finally use out of to the box to download the file

curlCall=" curl 'http://www_sitex_org/descarga.jsf' -H '...allCurlRequest....' > file.xml"
subprocess.call(curlCall, shell=True)
Respondent: hkeyland

The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 .

Leave a Reply

Your email address will not be published.