Web Scraping With Golang
Today, we’re looking at how you can build your first web application in Go, so open up your IDEs and let’s get started.
GoLang Web App Basic Setup
We’ll have to import net/http, and setup our main function with placeholders.
GoLang web application example. We can use net/http to create a web app in Go. ResponseWriter can write HTML output. HTML template and Redis example.
http.HandleFunc is a function that handles the paths in a url. For example http://www.golangdocs.com/2020/08/23.
Resolving the Complexities of Web Scraping with Python Picking the right tools, libraries, and frameworks. First and foremost, I can't stress enough the utility of browser tools for visual inspection. Effectively planning our web scraping approach upfront can probably save us hours of head scratching in advance. Scraping framework for extracting the data you need from websites, used for a wide range of applications, like data mining, data processing or archiving. In this workshop we will be looking at web scraping and how easily it can be done in Golang. Liam Hampton Liam is a Developer Advocate at IBM, working within the Developer Ecosystems Group. The first step to web scraping is being able to make an HTTP request. Let's look a very basic HTTP GET request and how to check the response code and view the content. Note the default timeout of an HTTP request using the default transport is forever. // makehttprequest.go.
- Here, the index page is linked to the homepage of our site.
- ListenAndServe listens to the port in the quotes, which is 8000. Once we run our web app, you can find it at localhost:8000.
Next, we need to configure the index page, so let’s create that:
Similar to if you’ve ever worked on Django, our function for index page takes input as a request to a url, and then responds with something. Replace the inside of the index_page function with anything of your choice (the w implies we want to write something), say,
Save this file as “webApp.go”, and we can run the following command in the terminal:
The following page comes up at localhost:8000 –
ResponseWriter Output HTML too
With Golang ResponseWriter, it is possible to directly format the text using HTML tags.
and that gives us the desired output:
This is still just one page, but say you wanted to make your site so that it will not return an error when you type localhost:8000/something_else.
Let’s code for that !
Output:
Voila !
Gorilla Mux for ease of web app development
Let me introduce you to a package named Gorilla Mux, which will make your web development much easier. So first, let’s install it using go get in the terminal.
We’ll do a few changes to our above code and use gorilla mux router instance instead of our indexHandler:
GoLang web application HTML templates
The hard coded design is quite plain and unimpressive. There is a method to rectify that – HTML templates. So let’s create another folder called “templates”, which will contain all the page designs.
We’ll also add a new file here called “index.html”, and add something simple:
Let’s switch back to our main .go file, and import our “html/template” package. Since our templates must be accessible from all handlers, let’s convert it to a global object:
Now we need to tell golang to parse our index.html for the template design and instantiate into our templates object:
Then modify the indexPage handler to contain:
And now if we run it, we’ll have exactly what we wanted.
Using Redis with Go web app
As a brief introduction to Redis, which we’ll be using as our database, they describe themselves best:
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams.
https://redis.io/So first download and install Redis: https://redis.io/download
Import the go-redis package and declare a global object:
Instantiate the redis client in main function:
and we need to grab some data from the redis server:
and then render into the index.html file:
We’re done configuring our html, which will take the elements from the comments array in our redis client, and place them in our web app.
So now we can open our command line and type in redis-cli to enter the redis shell, where we can push comments into the empty array:
Then if we run our app, you can see that it is now fetching the comments from the server. It would be able to do the same for, say, an AWS server.
Ending Notes
Making a web application can take anywhere from a few days to a few months depending on the complexity of the application. For every button or functionality, there is help in the official Golang documentation, so definitely check that out.
References
In general programming interfaces are contracts that have a set of functions to be implemented to fulfill that contract. Go is no different. Go has great support for interfaces and they are implemented in an implicit way. They allow polymorphism in Go. In this post, we will talk about interfaces, what they are, and how they can be used.
What is an Interface?
An interface is an abstract concept which enables polymorphism in Go. A variable of that interface can hold the value that implements the type. Type assertion is used to get the underlying concrete value as we will see in this post.
Declaring an interface in GoLang
An interface is declared as a type. Here is the declaration that is used to declare an interface.
type interfaceName interface{}
Zero-value of an interface
The zero value of an interface is nil. That means it holds no value and type. The code below shows that.
The empty interface in Go
An interface is empty if it has no functions at all. An empty interface holds any type. That’s why it is extremely useful in many cases. Below is the declaration of an empty interface.
var i interface{}
Implementing an interface in GoLang
An interface is implemented when the type has implemented the functions of the interface. Here is an example showing how to implement an interface.
Golang Web Api
Implementing multiple interfaces in Go
Multiple interfaces can be implemented at the same time. If all the functions are all implemented then the type implements all the interfaces. Below the type, the bird type implements both the interfaces by implementing the functions.
Composing interfaces together
Interfaces can be composed together. The composition is one of the most important concepts in software development. When multiple interfaces are implemented then the type has performed composition. This is really helpful where polymorphism is needed.
Values in an interface
Interface values have a concrete value and a dynamic type.
In the code above chirper is of type Bird but has a concrete value of {Chirpir}.
Type assertion using the interface
Type assertion is a way to get the underlying value an interface holds. This means if an interface variable is assigned a string then the underlying value it holds is the string. Here is an example showing how to use type assertion using interfaces.
Type switch using an interface
Type switches are an extremely similar control structure like the switch-cases, the only difference is here the interface type is used to switch between different conditions.
Equality of interface values
The interface values can be equal if any of the conditions shown below are true.
- They both are nil.
- They have the same underlying concrete values and the same dynamic type.
Golang Web Service Example
Web Scraping With Golang Design
Using interfaces with functions
Interfaces can be passed to functions just like any other type. Here is an example showing the usage of the interface with functions. A great advantage when using an interface is that it allows any type of argument as we can see in this code below.
Web Scraper Golang
Uses of an interface
Interfaces are used in Go where polymorphism is needed. In a function where multiple types can be passed an interface can be used. Interfaces allow Go to have polymorphism.
Interfaces are a great feature in Go and should be used wisely.