Perl: Template::Extract, Template::Generate and WWW:Mechanize
Translation of the original in Catalan. Read it in: CatalàCastellano

Normally I never talk about Perl libraries, although a good many years ago I used it quite a lot. Obviously PHP3 still existed and Python wasn't even a dream. In fact, I can't deny Perl's super power, the only problem I find is that it has such a confusing, complex or dense syntax, let's put it however we like, that if you leave it for a while afterwards you remember almost nothing. But I didn't want to comment on the difficulties I have remembering what I knew about Perl in this article.
I simply wanted to take the opportunity to reference a couple of articles by BrainStorm that talk about some tools that will change my life the next time I have to program a web spidering tool. In fact, until now I called them robots, basically they were tools that served to build databases for me from websites that had very complete information. For example, websites with collections of features of thousands of mobile phones.
The references to the articles:
The libraries BrainStorm tells us about:
- Template::Extract we build a template and by applying it to an HTML document it lets us obtain the data, without the tags. In fact, if we want we can even choose what information we want to extract, we don't have to extract it all. Specifically the templates are built according to the syntax called TT2.
- Template::Generate builds standard templates from an HTML we pass to it. So afterwards we only need to modify the template in order to adjust it to the needs we may have.
- WWW::Mechanize we can browse through a web page from inside a perl object. It's great because it lets us reach the data we're interested in obtaining completely transparently. For example, we connect to a website, register, go to the restricted area, click on a section of the website and get the documentation we wanted inside the perl object. Now if we want we can apply a Template::Extract to it, for example.
I'd say it's the perfect trilogy for generating tools that let us extract information from HTML documents. Another application, which is the one mentioned in the second article I referenced, is the creation of RSS summaries of pages that don't have that functionality.
2 comments
Comments from the old blog, as they were written. Comments are closed.
brainstorm
Ostres, si que t'han calat fons els meus articles, me'n alegro que et siguin tant útils :-)
—
Si et digues les hores que m'he passat programat web-spiders d'aquest amb PHP et pensaries que et prenc el pel. Per temes de feina he hagut d'omplir BBDD amb més de 10.000 registres amb aquesta tècnica. No et pots ni imaginar els mals de caps que he agafat amb el tema.
Per això m'ha agradat tan trobar aquesta alternativa en Perl, ara el proper cop que calgui fer això ja sé com sortir-me'n sense patir tan. Tot i que si em deixes escollir preferiria no haver-ho de fer més.