Curl To Python Requests



Introduction

In this article I am going to add ‘cURL’ and a Python library to the options for using these APIs, in either building clients or for testing REST API endpoints. API Keys There are a number of different access levels for users of an account depending on the level of read/write access of the key they have.

CURL code convert to some programming languages. Thanks for curlconverter! Currently supported languages: go, python, node, php, r, strest, rust, elixir, dart, json, ansible, matlab, etc. Install curl2 pip3 install -U curl2 How to use Python curl2 -c 'curl OutPut; import requests response = requests. Get ('https://leesoar. Curltorequests curltorequests is a Python module that converts cURL commands into equivalent Python code using the requests library. Many cURL features are unsupported (it's a very complex piece of software), but curltorequests is far more complete than any of the other cURL converting utilities I've found.

  • The GraphQL query. For this example, I'm using the space from my Serverless SuperHero website.
  • Kite is a free autocomplete for Python developers. Code faster with the Kite plugin for your code editor, featuring Line-of-Code Completions and cloudless processing.

If you’re using Elasticsearch to store data, you know that making cURL requests is a common way to communicate with your Elasticsearch cluster. Python’s requests library can be used to make cURL requests to an Elasticsearch cluster, making it easy to communicate with Elasticsearch from a Python script. In this step-by-step tutorial, we’ll explain how to make cURL requests to Elasticsearch in Python.

Prerequisites

Before we attempt to make any curl requests in Python, it’s important to make sure certain prerequisites are in place. For this task, a few key system requirements include:

  • You’ll need to have an Elasticsearch cluster running. The examples shown in this article work best with Elasticsearch version 6.0 or newer.

  • Python must be installed on the system or server where the Elasticsearch cluster is running. The examples in this article were written with Python 3 in mind, as Python 2 is scheduled for deprecation.

  • It’s helpful to have some experience with Python syntax and be somewhat familiar with structuring and creating cURL requests.

  • The elasticsearch Python library for the low-level Elasticsearch client must be installed for Python.

Install the Python modules for Elasticsearch and Requests

If you don’t already have the elasticsearch and requests modules installed, you’ll need to use Python’s PIP package manager to install them before proceeding:

Python 3 uses the command pip3 to install PIP packages:

pip3 install elasticsearch
pip3 install requests

If you’re running Python 2, simply use pip to install them:

Create the Python script used to make cURL requests

Once the necessary modules have been installed, create a directory if you haven’t done so already. You’ll also need to create a Python script that will be used to make the cURL requests. You can name them whatever you want, but our example will feature a directory named elastic-python and a script named curl_python.py:

sudomkdir elastic-python
sudotouch curl_python.py

Curl To Python Requests Pdf

Set up the Python script and import the Elasticsearch and Requests libraries:

Now it’s time to edit the Python script that you’ve just created. You can do this using the terminal-based text editor nano:

At the beginning of the script, you can add the line shown below to guide the system’s interpreter to the correct Python $PATH:

#!/usr/bin/env python3
#-*- coding: utf-8 -*-

Next, we’ll import a few Python libraries needed to make the requests:

Curl To Python Requests
from elasticsearch import Elasticsearch
importlogging# for debugging purposes
import requests
# import 'json' to convert strings to JSON format
import json

If you’d like to debug the cURL requests made to Elasticsearch, you can use the http.client module (or httplib in Python 2). This module returns useful information about cURL requests, such as the header used and the HTTP response code from the request:

# import the correct library depending on the Python version
try:
# Python 2
importhttplibas http_client
except(ImportError, ModuleNotFoundError)as error:
# Python 3
print('Python 3: Importing http.client:', error,'n')
import http.clientas http_client
# set the debug level
http_client.HTTPConnection.debuglevel=1
# initialize the logging to have the debugger return information
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
# store the DEBUG information in a 'requests_log' variable
requests_log =logging.getLogger('requests.packages.urllib3')
requests_log.setLevel(logging.DEBUG)
requests_log.propagate=True

The basic structure of a Requests call to Elasticsearch

Curl

Before we look at the next section of code, let’s take a moment to discuss the basic structure of a requests call to Elasticsearch. The method called from the Python requests class will depend on which HTTP verb is used. For example, you can make a GET request by calling requests.get() and a POST request by calling the requests.post() method, and so on.

Calling a method of Python’s Requests class:

''
make this Elasticsearch cURL request in Python:
curl -XGET localhost:9200/_cat/indices?v
''

# use a Python tuple to pass header options to the request call
param =(('v','),)# '-v' is for --verbose
# call the class's method to get an HTTP response model
resp = requests.get('http://localhost:9200/_cat/indices', params=param)
# the call will return a `requests.models.Response` object

After adding the code shown above, save the script, and then execute it in a terminal using the command python3 (if the elasticsearch module was installed using pip3), or the python command for Python 2:

NOTE: For Python 2 use: python curl_python.py

Running the script will produce the same result as making the following cURL request:

curl -XGET localhost:9200/_cat/indices?v

Print the HTTP response from the Requests call:

With just a few more lines of code in your script, you can print out the HTTP response returned from the Elasticsearch cluster:

print('nHTTP code:', resp.status_code,'-- response:', resp,'n')
print('dir:',dir(resp),'n')
print('response text:', resp.text)

Save the script again after adding this code, and run it in a terminal a second time. The response you receive should return information about the Elasticsearch cluster’s indices:

Create a Python function that makes cURL requests to Elasticsearch

Now that we understand the basics of the requests library, we can use it to create a function that’s designed with Elasticsearch in mind.

Avoid getting a 406 Content-Type error:

Elasticsearch requires that a header option be explicitly passed that specifies the request’s body type. For Elasticsearch requests, the body type will always be “JSON”.

In the Python request class, you can pass this header option as a parameter called header when making a request:

requests.get(uri, data=query, headers='content-type:application/json')
Curl To Python Requests

406 Content-Type error after making a query request to Elasticsearch that’s missing content-type:application/json in the request’s header:

Define a Python function for Elasticsearch cURL requests:

Here, we define a function to simplify calls made to the Requests class for Elasticsearch cURL requests. To be safe, the default HTTP verb is GET to avoid making unwanted changes to the Elasticsearch cluster:

Curl To Python Requests Calculator

Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# function for the cURL requests
def elasticsearch_curl(uri='http://localhost:9200/', json_body=', verb='get'):
# pass header option for content type if request has a
# body to avoid Content-Type error in Elasticsearch v6.0
headers ={
'Content-Type': 'application/json',
}
try:
# make HTTP verb parameter case-insensitive by converting to lower()
if verb.lower()'get':
resp = requests.get(uri, headers=headers, data=json_body)
elif verb.lower()'post':
resp = requests.post(uri, headers=headers, data=json_body)
elif verb.lower()'put':
resp = requests.put(uri, headers=headers, data=json_body)
# read the text object string
try:
resp_text = json.loads(resp.text)
except:
resp_text = resp.text
# catch exceptions and print errors to terminal
exceptExceptionas error:
print('nelasticsearch_curl() error:', error)
resp_text = error
# return the Python dict of the request
print('resp_text:', resp_text)
return resp_text

This function breaks down all of the method calls for the request class and streamlines it in one call that uses more simplified parameters.

Here are a few examples that call the function to make cURL requests to the Elasticsearch cluster:

print('n')
response = elasticsearch_curl('http://localhost:9200/some_index')
print('RESPONSE 1:', response,'nn')
response = elasticsearch_curl('http://localhost:9200/_cat/indices')
print('RESPONSE 2:', response,'nn')
request_body =''
{
'settings' : {
'number_of_shards' : 3,
'number_of_replicas' : 2
}
}
''

response = elasticsearch_curl(
'http://localhost:9200/new_index',
verb='put',
json_body=request_body
)
print('RESPONSE 3:', response,'nn')

Curl To Python Requests Login

Conclusion

Making cURL requests to an Elasticsearch cluster is a simple and efficient way to communicate with Elasticsearch from a script. Python’s requests library can help automate the process of making cURL requests, which also results in cleaner, simpler code. With the step-by-step instructions included in this article, you’ll have no trouble making cURL requests to Elasticsearch using a Python script.

In this article, we looked at the example code one section at a time. Here’s the complete Python script, which includes the elasticsearch_curl() function, for making cURL requests to Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
#!/usr/bin/env python3
#-*- coding: utf-8 -*-
# import the Elasticsearch client library
from elasticsearch import Elasticsearch
# for debugging purposes
importlogging
# used to make HTTP requests to the Elasticsearch cluster
import requests
# import 'json' to convert strings to JSON format
import json
# catch import errors that arise from breaking version
# changes from Python 2 and 3
try:
# try Python 2
importhttplibas http_client
except(ImportError, ModuleNotFoundError)as error:
# try Python 3
print('Python 3: Importing http.client:', error,'n')
import http.clientas http_client
# set the debug level
http_client.HTTPConnection.debuglevel=1
# initialize the logging to have the debugger return information
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
# store the DEBUG information in a 'requests_log' variable
requests_log =logging.getLogger('requests.packages.urllib3')
requests_log.setLevel(logging.DEBUG)
requests_log.propagate=True
''
make this Elasticsearch cURL request in Python:
curl -XGET localhost:9200/_cat/indices?v
''

# use a Python tuple to pass header options to the request call
param =(('v','),)# '-v' is for --verbose
# call the class's method to get an HTTP response model
resp = requests.get('http://localhost:9200/_cat/indices', params=param)
# the call will return a `requests.models.Response` object
print('nHTTP code:', resp.status_code,'-- response:', resp,'n')
print('dir:',dir(resp),'n')
print('response text:', resp.text)
# function for the cURL requests
def elasticsearch_curl(uri='http://localhost:9200/', json_body=', verb='get'):
# pass header option for content type if request has a
# body to avoid Content-Type error in Elasticsearch v6.0
headers ={
'Content-Type': 'application/json',
}
try:
# make HTTP verb parameter case-insensitive by converting to lower()
if verb.lower()'get':
resp = requests.get(uri, headers=headers, data=json_body)
elif verb.lower()'post':
resp = requests.post(uri, headers=headers, data=json_body)
elif verb.lower()'put':
resp = requests.put(uri, headers=headers, data=json_body)
# read the text object string
try:
resp_text = json.loads(resp.text)
except:
resp_text = resp.text
# catch exceptions and print errors to terminal
exceptExceptionas error:
print('nelasticsearch_curl() error:', error)
resp_text = error
# return the Python dict of the request
print('resp_text:', resp_text)
return resp_text
print('n')
response = elasticsearch_curl('http://localhost:9200/some_index')
print('RESPONSE 1:', response,'nn')
response = elasticsearch_curl('http://localhost:9200/_cat/indices')
print('RESPONSE 2:', response,'nn')
request_body =''
{
'settings' : {
'number_of_shards' : 3,
'number_of_replicas' : 2
}
}
''

response = elasticsearch_curl(
'http://localhost:9200/new_index',
verb='put',
json_body=request_body
)
print('RESPONSE 3:', response,'nn')