Script to fetch Page Title from URL

Lucid_Chaotic · June 1, 2007, 3:43pm

Hi guys,

I urgently need a script that can run through a text file of URLs and traverse the links and generate each link’s respective page title.

So given a list of URLs as input, I need to get a list of webpage titles as the output.

Has anybody seen such a script or written one? I can write one myself but the need is rather urgent and thought I’d ask around first than spend a few hours on my own to write something.

TofiBaba · June 1, 2007, 7:08pm

Re: Script to fetch Page Title from URL

Would this application work? I know its not free but u said urgent:

http://www.spadixbd.com/extracturl/index.htm

CuR10u · June 2, 2007, 6:00am

Re: Script to fetch Page Title from URL

Python script chale ga bhai ??

Anyway, i have modified my parser script to do just the “titles” … i mean the title tag. It just parses simple string, not any special characters that may appear in the title. Those could be parsed as well; let me know if u face such titles.

You can download Python from Download Python | Python.org

Following script fetches titles from Google, Yahoo, and Hotmail. Those URLs are hard-coded in the “urls” list/array.

#!/usr/bin/python
import re
import urllib
import sgmllib

class Page_Parser(sgmllib.SGMLParser):

The parser class

def parse(self, s):
self.feed(s)
self.close()

def init(self, verbose=0):
sgmllib.SGMLParser.init(self, verbose)
# Initialize vars.
self.atTitle = 0
self.title = ''
Parsing the tags.

def start_title(self, attrs):
self.atTitle = 1
def end_title(self):
“end of div”

def handle_data(self, data):
if self.atTitle == 1:
    self.title += data
    self.atTitle = 0
#…End of Page_Parser class…

#…Starts here!..

urls = ‘http://www.yahoo.com’, ‘http://www.google.com’, ‘http://www.hotmail.com’]

count = 0
print ’
------Starts parsign title------

’

while count < len(urls):
try:
    page = urllib.urlopen(urls[count])

except:
    print '
Error!
’
sys.exit(-1)
page_top = page.read(1000)

pgParser = Page_Parser()
pgParser.parse(page_top)
print pgParser.title + "    ("+ urls[count] + ")"
pgParser.title = ''

page_top = ''

count += 1
print ’

------parsing is over.------
’

CuR10u · June 2, 2007, 6:05am

Re: Script to fetch Page Title from URL

My output is the following:

CuR10u · June 2, 2007, 7:28pm

Re: Script to fetch Page Title from URL

ah!, just noticed that vBulletin (gs) has removed all indentation of the script. Since python does not use {} for block code, it only relies on tabbed spaces. But as u see every statement in alligned to the left-most column, so it won’t run like that. If u want to run it, i will have to send the file prob. through email.

And i missed the point that you want to read URLs from a text file, so i have modified code and have also made it to dump the results into another text file as well.

Following is the change; + shows the added code, and dots are everything same as above.

[quote]

import sys
.
.
.
.
#…Starts here!..
urls = ]
url_file = open(‘urls.txt’)
urls_read = url_file.readlines()
url_file.close()
for s in urls_read:
```
urls.append(re.sub('
```

', ‘’, s))

#urls = ‘http://www.yahoo.com’, ‘http://www.google.com’, ‘http://www.hotmail.com’]

count = 0
print ’
------Starts parsign title------

’

output_file = open(‘titles.txt’, ‘wb’)
.
.
. # (everything same here)
.
.
pgParser.parse(page_top)
output = pgParser.title + " ("+ urls[count] + “)”
print output
output_file.write(output + ’
')
pgParser.title = ‘’
.
.
.
.
count += 1
output_file.close()
print ’

------parsing is over.------
’

Lucid_Chaotic · June 4, 2007, 2:25am

Re: Script to fetch Page Title from URL

Thanks Cur10u$... this looks like it'll work for me.

I ended up writing an Activex script to do this for me but it's only an interim solution. I'll see if I can use yours more formally in my project.

Thanks again!