Category Archives: Uncategorized

How to make an HTTP Post with the Requests Package in Python 3.4

by in Uncategorized | Leave a comment
python 3, HTTP Post, Ipython Notebook, Requests

A look from Ipython Notebook inverted.

Twitter @CodeDocta

Making an HTTP POST Request

 

import requests
from pprint import pprint



URL = 'http://127.0.0.1:5000/post'

REFERER  = 'http://127.0.0.1:5000/forms/post'
UA = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0'

HEADERS = {
    'Host': 'httpbin.org',
    'Referer': REFERER,
    'User-Agent': UA,
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate',
    'DNT': 1,
    'Content-Type': 'application/x-www-form-urlencoded'
    }
PARAMS = {
    'custname': 'Nick',
    'custtel': '777-867-5309',
    'custemail': 'noob@lala.com',
    'size': 'medium',
    'topping': 'onion',
    'delivery': '18:20',
    'comments': 'I woutld like extra peppers please.'
}

PROXY = 'proxy goes here'
PROXIES = {
  "http": "http://" + PROXY,
  "https": "http://" + PROXY,
    }

resp = requests.post(URL, data=PARAMS, headers=HEADERS, proxies=None)
resp.close()
resp.status_code



200

Cool we have our 200

This means it worked and we can do something else, like save all the inputs to a databease or file.

 

if resp.status_code == 200:
    pprint(resp.text)



('{\n'
 '  "args": {},\n'
 '  "data": "",\n'
 '  "files": {},\n'
 '  "form": {\n'
 '    "comments": "I woutld like extra peppers please.",\n'
 '    "custemail": "noob@lala.com",\n'
 '    "custname": "Nick",\n'
 '    "custtel": "777-867-5309",\n'
 '    "delivery": "18:20",\n'
 '    "size": "medium",\n'
 '    "topping": "onion"\n'
 '  },\n'
 '  "headers": {\n'
 '    "Accept": '
 '"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",\n'
 '    "Accept-Encoding": "gzip, deflate",\n'
 '    "Accept-Language": "en-US,en;q=0.5",\n'
 '    "Connection": "keep-alive",\n'
 '    "Content-Length": "148",\n'
 '    "Content-Type": "application/x-www-form-urlencoded",\n'
 '    "Dnt": "1",\n'
 '    "Host": "httpbin.org",\n'
 '    "Referer": "http://127.0.0.1:5000/forms/post",\n'
 '    "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) '
 'Gecko/20100101 Firefox/37.0"\n'
 '  },\n'
 '  "json": null,\n'
 '  "origin": "127.0.0.1",\n'
 '  "url": "http://httpbin.org/post"\n'
 '}\n')

Without pprint it looks a little messy, try it on an real webpage.

pprint = pretty print

Well, that’s it…

Ok, so I lied. Let’s do some other cool stuff.

If is not the way to go here we need some exception handling.

 

PARAMS = {
    'custname': 'Nick',
    'custtel': '777-867-5309',
    'custemail': 'noob@lala.com',
    'size': 'medium',
    'topping': 'mushroom',
    'delivery': '18:20',
    'comments': 'can you make it free?'
}

try:
    r = requests.post("http://httpbin.org/pos", data=PARAMS, proxies=PROXIES)
    print(r.text)
    
except Exception as postError:
    print('AHHH... Your end of the world message!!!')
    print(postError)
    
r.close()



AHHH... Your end of the world message!!!
HTTPConnectionPool(host='proxy goes here', port=80): Max retries exceeded with url: http://httpbin.org/pos (Caused by ProxyError('Cannot connect to proxy.', gaierror(11004, 'getaddrinfo failed')))

I suggest you read deeply into the Exception Handling documentatoin there are many and Exception is the catch all. At some point I will do a tut on this module.

 

PARAMS = {
    'custname': 'Nick',
    'custtel': '888-867-5309',
    'custemail': 'noob@lala.com',
    'size': 'medium',
    'topping': 'mushroom',
    'delivery': '18:20',
    'comments': 'can you make it free?'
}

try:
    r = requests.post("http://127.0.0.1:5000/post", data=PARAMS, proxies=PROXIES, timeout=5)
    print(r.text)
    
except Exception as postError:
    print('You can put whatever on top of the real error.')
    print(postError)
    
r.close()



You can put whatever on top of the real error.
HTTPConnectionPool(host='proxy goes here', port=80): Max retries exceeded with url: http://127.0.0.1:5000/post (Caused by ProxyError('Cannot connect to proxy.', gaierror(11004, 'getaddrinfo failed')))

As you can see I did not change the proxy to an real proxy however you see the error is quite telling.

You can see some defaults there like port, max redirects and I snuck in a timeout on you. There are 2 time outs one for the read and one for the server connection. Now that you are awhere you can go see more about them in the Requests documentation. It is done well so don’t be scared.

Timeouts http://docs.python-requests.org/en/latest/user/advanced/#timeouts and http://docs.python-requests.org/en/latest/user/quickstart/#timeouts

Requests has its own exception handling too http://docs.python-requests.org/en/latest/user/quickstart/#errors-and-exceptions

Let’s see how to unpack a list into variables. I do this because I can read without thinking about do I have the correct list item when I try to debug.

 

pList = ['77.888.45.80.8080', 'Jon Doe', '555-867-5309', 'Jon.Doe@Amail.com', 'large', 'mushroom', '12:00', 'Make sure that coke is a diet coke!!']
PROXY, CUST, PHONE, EMAIL, SIZE, TOP, TIME, COMM = pList

pList



['77.888.45.80.8080',
 'Jon Doe',
 '555-867-5309',
 'Jon.Doe@Amail.com',
 'large',
 'mushroom',
 '12:00',
 'Make sure that coke is a diet coke!!']

Now we can use dynamic parameters…

 

PROXIES = PROXY
PARAMS = {
    'custname': CUST,
    'custtel': PHONE,
    'custemail': EMAIL,
    'size': SIZE,
    'topping': TOP,
    'delivery': TIME,
    'comments': COMM
}

try:
    r = requests.post("http://127.0.0.1:5000/post", data=PARAMS, proxies=None, timeout=5)
    print(r.text)
    
except Exception as postError:
    print('You can put whatever on top of the real error.')
    print(postError)
    
r.close()



{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "comments": "Make sure that coke is a diet coke!!",
    "custemail": "Jon.Doe@Amail.com",
    "custname": "Jon Doe",
    "custtel": "555-867-5309",
    "delivery": "12:00",
    "size": "large",
    "topping": "mushroom"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "keep-alive",
    "Content-Length": "162",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "127.0.0.1:5000",
    "User-Agent": "python-requests/2.6.2 CPython/3.4.3 Windows/7"
  },
  "json": null,
  "origin": "127.0.0.1",
  "url": "http://127.0.0.1:5000/post"
}

When we are done with or memory slots you can delete them with the “del” keyword…

 

PARAMS



{'comments': 'Make sure that coke is a diet coke!!',
 'custemail': 'Jon.Doe@Amail.com',
 'custname': 'Jon Doe',
 'custtel': '555-867-5309',
 'delivery': '12:00',
 'size': 'large',
 'topping': 'mushroom'}


del PARAMS

This way we don’t have memory leaks and build up. Use this on variables and iterables when you no longer need them.

 

PARAMS



---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-15-6908596e41fe> in <module>()
----> 1 PARAMS

NameError: name 'PARAMS' is not defined

Twitter @CodeDocta