back to listing index

Smarp/GoOse

[web search]
Original source (github.com)
Tags: goose
Clipped on: 2016-06-11

Skip to content
Html Content / Article Extractor in Golang
HTML Go Makefile
Pull request Compare This branch is 4 commits ahead of advancedlogic:master.
sites trimmed img url. return image url even if it's invalid. tests fixed 4 months ago
.gitignore DEV-4442 ignore target dir, *.test and *.prof files 7 months ago
.travis.yml improved coverage report and add support for travis-ci and coveralls 7 months ago
LICENSE Initial commit 2 years ago
Makefile improved coverage report and add support for travis-ci and coveralls 7 months ago
README.md improved coverage report and add support for travis-ci and coveralls 7 months ago
article.go DEV-4442 extract article links 7 months ago
cleaner.go DEV-4442 extract article links 7 months ago
configuration.go extract user agent to const 9 months ago
crawler.go Merge branch 'orig-master' 4 months ago
crawler_test.go trimmed img url. return image url even if it's invalid. tests fixed 4 months ago
doc.go Create doc.go 2 years ago
extractor.go Merge branch 'orig-master' 4 months ago
goose.go Change crawl to return error instead of panicking. 6 months ago
goose.json Create goose.json 2 years ago
helpers.go golint 9 months ago
images.go trimmed img url. return image url even if it's invalid. tests fixed 4 months ago
outputformatter.go DEV-4442 extract article links 7 months ago
parser.go golint 9 months ago
stopwords.go golint 9 months ago
utils.go golint 9 months ago
videos.go golint 9 months ago
wordstats.go golint 9 months ago

README.md

GoOse

Build Status Image (Asset 3/3) alt=

Html Content / Article Extractor in Golang

This is a golang port of "Goose" originaly licensed to Gravity.com under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership.

Golang port was written by Antonio Linari

Gravity.com licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

INSTALL

go get github.com/advancedlogic/GoOse

HOW TO USE IT

package main

import (
    "github.com/advancedlogic/GoOse"
)

func main() {
    g := goose.New()
    article := g.ExtractFromURL("http://edition.cnn.com/2012/07/08/opinion/banzi-ted-open-source/index.html")
    println("title", article.Title)
    println("description", article.MetaDescription)
    println("keywords", article.MetaKeywords)
    println("content", article.CleanedText)
    println("url", article.FinalURL)
    println("top image", article.TopImage)
}

Development - Getting started

This application is written in GO language, please refere to the guides in https://golang.org for getting started.

This project include a Makefile that allows you to test and build the project with simple commands. To see all available options:

make help

To build the project

make build

Before committing the code, please check if it passes all tests using

make qa

TODO

  • better organize code
  • add comments and godoc
  • improve "xpath" like queries
  • add other image extractions techniques (imagemagick)

THANKS TO

normal@Martin Angers for goquery
@Fatih Arslan for set
GoLang team for the amazing language and net/html
normal