back to listing index

suzuken/GoOse

[web search]
Original source (github.com)
Tags: golang go goose
Clipped on: 2016-06-11

Skip to content
Html Content / Article Extractor in Golang
HTML Go Makefile
Pull request Compare This branch is 30 commits ahead of advancedlogic:master.
goose-server remove unused field from Article struct 4 months ago
sites DEV-4442 new test cases 7 months ago
.gitignore DEV-4442 ignore target dir, *.test and *.prof files 7 months ago
.travis.yml improved coverage report and add support for travis-ci and coveralls 7 months ago
LICENSE Initial commit 2 years ago
Makefile testing only at top directory 4 months ago
README.md add GoDoc badge on README 4 months ago
article.go remove unused field from Article struct 4 months ago
cleaner.go crawler: use io.Reader for extracting content 4 months ago
configuration.go extract user agent to const 9 months ago
crawler.go crawler: use io.Reader for extracting content 4 months ago
crawler_test.go crawler: use io.Reader for extracting content 4 months ago
doc.go fix misspell, docs, and some variable names. 4 months ago
extractor.go crawler: use io.Reader for extracting content 4 months ago
goose.go crawler: use io.Reader for extracting content 4 months ago
goose.json Create goose.json 2 years ago
images.go crawler: use io.Reader for extracting content 4 months ago
outputformatter.go fix misspell, docs, and some variable names. 4 months ago
parser.go use goquery from the original repository 4 months ago
stopwords.go use MultilangTokenizer for tokenizing a given string 4 months ago
stopwords_test.go add tokenizer for switching tokenizer in each language 4 months ago
tokenizer.go fix misspell, docs, and some variable names. 4 months ago
tokenizer_test.go add tokenizer for switching tokenizer in each language 4 months ago
videos.go crawler: use io.Reader for extracting content 4 months ago
wordstats.go golint 9 months ago

README.md

GoOse

Image (Asset 3/6) alt= Image (Asset 4/6) alt= Image (Asset 5/6) alt= Image (Asset 6/6) alt=

Html Content / Article Extractor in Golang

This is a golang port of "Goose" originaly licensed to Gravity.com under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership.

Golang port was written by Antonio Linari

Gravity.com licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

INSTALL

go get github.com/suzuken/GoOse

HOW TO USE IT

package main

import (
    "github.com/suzuken/GoOse"
)

func main() {
    g := goose.New()
    article, err := g.ExtractFromURL("http://edition.cnn.com/2012/07/08/opinion/banzi-ted-open-source/index.html")
    if err != nil {
        panic(err)
    }
    println("title", article.Title)
    println("description", article.MetaDescription)
    println("keywords", article.MetaKeywords)
    println("content", article.CleanedText)
    println("url", article.FinalURL)
    println("top image", article.TopImage)
}

Development - Getting started

This application is written in GO language, please refere to the guides in https://golang.org for getting started.

This project include a Makefile that allows you to test and build the project with simple commands. To see all available options:

make help

To build the project

make build

Before committing the code, please check if it passes all tests using

make qa

TODO

  • better organize code
  • add comments and godoc
  • improve "xpath" like queries
  • add other image extractions techniques (imagemagick)

THANKS TO

normal@Martin Angers for goquery
@Fatih Arslan for set
GoLang team for the amazing language and net/html
normal