Curl To Python Requests
Introduction
In this article I am going to add ‘cURL’ and a Python library to the options for using these APIs, in either building clients or for testing REST API endpoints. API Keys There are a number of different access levels for users of an account depending on the level of read/write access of the key they have.
CURL code convert to some programming languages. Thanks for curlconverter! Currently supported languages: go, python, node, php, r, strest, rust, elixir, dart, json, ansible, matlab, etc. Install curl2 pip3 install -U curl2 How to use Python curl2 -c 'curl OutPut; import requests response = requests. Get ('https://leesoar. Curltorequests curltorequests is a Python module that converts cURL commands into equivalent Python code using the requests library. Many cURL features are unsupported (it's a very complex piece of software), but curltorequests is far more complete than any of the other cURL converting utilities I've found.
- The GraphQL query. For this example, I'm using the space from my Serverless SuperHero website.
- Kite is a free autocomplete for Python developers. Code faster with the Kite plugin for your code editor, featuring Line-of-Code Completions and cloudless processing.
If you’re using Elasticsearch to store data, you know that making cURL requests is a common way to communicate with your Elasticsearch cluster. Python’s requests
library can be used to make cURL requests to an Elasticsearch cluster, making it easy to communicate with Elasticsearch from a Python script. In this step-by-step tutorial, we’ll explain how to make cURL requests to Elasticsearch in Python.
Prerequisites
Before we attempt to make any curl requests in Python, it’s important to make sure certain prerequisites are in place. For this task, a few key system requirements include:
You’ll need to have an Elasticsearch cluster running. The examples shown in this article work best with Elasticsearch version 6.0 or newer.
Python must be installed on the system or server where the Elasticsearch cluster is running. The examples in this article were written with Python 3 in mind, as Python 2 is scheduled for deprecation.
It’s helpful to have some experience with Python syntax and be somewhat familiar with structuring and creating cURL requests.
The
elasticsearch
Python library for the low-level Elasticsearch client must be installed for Python.
Install the Python modules for Elasticsearch and Requests
If you don’t already have the elasticsearch
and requests
modules installed, you’ll need to use Python’s PIP package manager to install them before proceeding:
Python 3 uses the command pip3
to install PIP packages:
pip3 install elasticsearch pip3 install requests |
If you’re running Python 2, simply use pip
to install them:
Create the Python script used to make cURL requests
Once the necessary modules have been installed, create a directory if you haven’t done so already. You’ll also need to create a Python script that will be used to make the cURL requests. You can name them whatever you want, but our example will feature a directory named elastic-python
and a script named curl_python.py
:
sudomkdir elastic-python sudotouch curl_python.py |
Curl To Python Requests Pdf
Set up the Python script and import the Elasticsearch and Requests libraries:
Now it’s time to edit the Python script that you’ve just created. You can do this using the terminal-based text editor nano
:
At the beginning of the script, you can add the line shown below to guide the system’s interpreter to the correct Python $PATH
:
#!/usr/bin/env python3 #-*- coding: utf-8 -*- |
Next, we’ll import a few Python libraries needed to make the requests:
from elasticsearch import Elasticsearch importlogging# for debugging purposes import requests # import 'json' to convert strings to JSON format import json |
If you’d like to debug the cURL requests made to Elasticsearch, you can use the http.client
module (or httplib
in Python 2). This module returns useful information about cURL requests, such as the header used and the HTTP response code from the request:
# import the correct library depending on the Python version try: # Python 2 importhttplibas http_client except(ImportError, ModuleNotFoundError)as error: # Python 3 print('Python 3: Importing http.client:', error,'n') import http.clientas http_client # set the debug level http_client.HTTPConnection.debuglevel=1 # initialize the logging to have the debugger return information logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) # store the DEBUG information in a 'requests_log' variable requests_log =logging.getLogger('requests.packages.urllib3') requests_log.setLevel(logging.DEBUG) requests_log.propagate=True |
The basic structure of a Requests call to Elasticsearch
Before we look at the next section of code, let’s take a moment to discuss the basic structure of a requests
call to Elasticsearch. The method called from the Python requests
class will depend on which HTTP verb is used. For example, you can make a GET
request by calling requests.get()
and a POST
request by calling the requests.post()
method, and so on.
Calling a method of Python’s Requests class:
'' make this Elasticsearch cURL request in Python: curl -XGET localhost:9200/_cat/indices?v '' # use a Python tuple to pass header options to the request call param =(('v','),)# '-v' is for --verbose # call the class's method to get an HTTP response model resp = requests.get('http://localhost:9200/_cat/indices', params=param) # the call will return a `requests.models.Response` object |
After adding the code shown above, save the script, and then execute it in a terminal using the command python3
(if the elasticsearch
module was installed using pip3
), or the python
command for Python 2:
NOTE: For Python 2 use: python curl_python.py
Running the script will produce the same result as making the following cURL request:
curl -XGET localhost:9200/_cat/indices?v |
Print the HTTP response from the Requests call:
With just a few more lines of code in your script, you can print out the HTTP response returned from the Elasticsearch cluster:
print('nHTTP code:', resp.status_code,'-- response:', resp,'n') print('dir:',dir(resp),'n') print('response text:', resp.text) |
Save the script again after adding this code, and run it in a terminal a second time. The response you receive should return information about the Elasticsearch cluster’s indices:
Create a Python function that makes cURL requests to Elasticsearch
Now that we understand the basics of the requests
library, we can use it to create a function that’s designed with Elasticsearch in mind.
Avoid getting a 406 Content-Type error:
Elasticsearch requires that a header option be explicitly passed that specifies the request’s body type. For Elasticsearch requests, the body type will always be “JSON”.
In the Python request
class, you can pass this header option as a parameter called header
when making a request:
requests.get(uri, data=query, headers='content-type:application/json') |
406 Content-Type error after making a query request to Elasticsearch that’s missing content-type:application/json in the request’s header:
Define a Python function for Elasticsearch cURL requests:
Here, we define a function to simplify calls made to the Requests class for Elasticsearch cURL requests. To be safe, the default HTTP verb is GET
to avoid making unwanted changes to the Elasticsearch cluster:
Curl To Python Requests Calculator
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | # function for the cURL requests def elasticsearch_curl(uri='http://localhost:9200/', json_body=', verb='get'): # pass header option for content type if request has a # body to avoid Content-Type error in Elasticsearch v6.0 headers ={ 'Content-Type': 'application/json', } try: # make HTTP verb parameter case-insensitive by converting to lower() if verb.lower()'get': resp = requests.get(uri, headers=headers, data=json_body) elif verb.lower()'post': resp = requests.post(uri, headers=headers, data=json_body) elif verb.lower()'put': resp = requests.put(uri, headers=headers, data=json_body) # read the text object string try: resp_text = json.loads(resp.text) except: resp_text = resp.text # catch exceptions and print errors to terminal exceptExceptionas error: print('nelasticsearch_curl() error:', error) resp_text = error # return the Python dict of the request print('resp_text:', resp_text) return resp_text |
This function breaks down all of the method calls for the request
class and streamlines it in one call that uses more simplified parameters.
Here are a few examples that call the function to make cURL requests to the Elasticsearch cluster:
print('n') response = elasticsearch_curl('http://localhost:9200/some_index') print('RESPONSE 1:', response,'nn') response = elasticsearch_curl('http://localhost:9200/_cat/indices') print('RESPONSE 2:', response,'nn') request_body ='' { 'settings' : { 'number_of_shards' : 3, 'number_of_replicas' : 2 } } '' response = elasticsearch_curl( 'http://localhost:9200/new_index', verb='put', json_body=request_body ) print('RESPONSE 3:', response,'nn') |
Curl To Python Requests Login
Conclusion
Making cURL requests to an Elasticsearch cluster is a simple and efficient way to communicate with Elasticsearch from a script. Python’s requests
library can help automate the process of making cURL requests, which also results in cleaner, simpler code. With the step-by-step instructions included in this article, you’ll have no trouble making cURL requests to Elasticsearch using a Python script.
In this article, we looked at the example code one section at a time. Here’s the complete Python script, which includes the elasticsearch_curl()
function, for making cURL requests to Elasticsearch:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | #!/usr/bin/env python3 #-*- coding: utf-8 -*- # import the Elasticsearch client library from elasticsearch import Elasticsearch # for debugging purposes importlogging # used to make HTTP requests to the Elasticsearch cluster import requests # import 'json' to convert strings to JSON format import json # catch import errors that arise from breaking version # changes from Python 2 and 3 try: # try Python 2 importhttplibas http_client except(ImportError, ModuleNotFoundError)as error: # try Python 3 print('Python 3: Importing http.client:', error,'n') import http.clientas http_client # set the debug level http_client.HTTPConnection.debuglevel=1 # initialize the logging to have the debugger return information logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) # store the DEBUG information in a 'requests_log' variable requests_log =logging.getLogger('requests.packages.urllib3') requests_log.setLevel(logging.DEBUG) requests_log.propagate=True '' make this Elasticsearch cURL request in Python: curl -XGET localhost:9200/_cat/indices?v '' # use a Python tuple to pass header options to the request call param =(('v','),)# '-v' is for --verbose # call the class's method to get an HTTP response model resp = requests.get('http://localhost:9200/_cat/indices', params=param) # the call will return a `requests.models.Response` object print('nHTTP code:', resp.status_code,'-- response:', resp,'n') print('dir:',dir(resp),'n') print('response text:', resp.text) # function for the cURL requests def elasticsearch_curl(uri='http://localhost:9200/', json_body=', verb='get'): # pass header option for content type if request has a # body to avoid Content-Type error in Elasticsearch v6.0 headers ={ 'Content-Type': 'application/json', } try: # make HTTP verb parameter case-insensitive by converting to lower() if verb.lower()'get': resp = requests.get(uri, headers=headers, data=json_body) elif verb.lower()'post': resp = requests.post(uri, headers=headers, data=json_body) elif verb.lower()'put': resp = requests.put(uri, headers=headers, data=json_body) # read the text object string try: resp_text = json.loads(resp.text) except: resp_text = resp.text # catch exceptions and print errors to terminal exceptExceptionas error: print('nelasticsearch_curl() error:', error) resp_text = error # return the Python dict of the request print('resp_text:', resp_text) return resp_text print('n') response = elasticsearch_curl('http://localhost:9200/some_index') print('RESPONSE 1:', response,'nn') response = elasticsearch_curl('http://localhost:9200/_cat/indices') print('RESPONSE 2:', response,'nn') request_body ='' { 'settings' : { 'number_of_shards' : 3, 'number_of_replicas' : 2 } } '' response = elasticsearch_curl( 'http://localhost:9200/new_index', verb='put', json_body=request_body ) print('RESPONSE 3:', response,'nn') |