Showing posts with label crawler. Show all posts
Showing posts with label crawler. Show all posts

Saturday, March 4, 2017

URL Encoding and Decoding using Java

It is common requirement to implement URL encoding and decoding in Java while creating crawlers or downloaders. This post focus on creating modules for encoding and decoding of the passed url using Java.

Language Used:
Java

Git Location:
https://github.com/csanuragjain/extra/tree/master/EncodeDecodeURL

Program:
main method:
 public static void main(String[] args) {  
           // TODO Auto-generated method stub  
           String url="https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%252Fmp4%26pl%3D21%26itag%3D22%26\u0026itag=43\u0026type=video%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22\u0026quality=medium";  
           String url2="https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs=\"vp8.0, vorbis\"&quality=medium";  
           String decodeURL = decode(url);  
           System.out.println("Decoded URL: "+decodeURL);  
           String encodeURL = encode(url2);  
           System.out.println("Encoded URL2: "+encodeURL);  
      }  

How it works:
1) url is the variable having encoded url which we want to decode
2) url2 is the variable having an url which we want to encode
3) We call the decode method which decodes the url and then print the same.
4) We call the encode method which encodes the url2 and then print the same.

encode method:
      public static String encode(String url)  
      {  
                try {  
                     String encodeURL=URLEncoder.encode( url, "UTF-8" );  
                     return encodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while encoding" +e.getMessage();  
                }  
      }  

How it works:
1) We use the encode method of a predefined java class named URLEncoder
2) encode method of URLEncoder takes 2 arguments
3) 1st argument defines the url to be encoded
4) 2nd argument defines the encoding scheme to be used
5) After encoding the resulting encoded url is returned

decode method:
      public static String decode(String url)  
      {  
                try {  
                     String prevURL="";  
                     String decodeURL=url;  
                     while(!prevURL.equals(decodeURL))  
                     {  
                          prevURL=decodeURL;  
                          decodeURL=URLDecoder.decode( decodeURL, "UTF-8" );  
                     }  
                     return decodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while decoding" +e.getMessage();  
                }  
      }  

How it works:
1) Since same url can be encoded multiple times so we need to decode until url cannnot be decoded further.
2) For eg: "video%252Fmp4" is result of 2 encoding. On decoding it once we get "video%2Fmp4". Now url need to be further decoded so when we apply decoding again we get "video/mp4", which is the result.
3) We use the decode method of a predefined java class named URLDecoder
4) decode method of URLDecoder takes 2 arguments
5) 1st argument defines the url to be decoded
6) 2nd argument defines the decoding scheme to be used
7) After decoding the resulting decoded url is returned
8) We create 2 variables prevURL which is empty and decodeURL which contain the url to be decoded.
 Variable State:  
 prevURL = ""  
 decodeURL ="somethingvideo%252Fmp4"  
9) We create an iteration which runs until prevURL!=decodeURL
10) Now we update prevURL to decodeURL and update decodeURL with the decoded value of the url passed.
 Variable State:  
 prevURL = "somethingvideo%252Fmp4"  
 decodeURL ="somethingvideo%2Fmp4"  
11) Since prevURL!=decodeURL so Step 10 runs again
 Variable State:  
 prevURL = "somethingvideo%2Fmp4"  
 decodeURL ="somethingvideo/mp4"  
12) Since prevURL!=decodeURL so Step 10 runs again
 Variable State:  
 prevURL = "somethingvideo/mp4"  
 decodeURL ="somethingvideo/mp4"  
13) Since prevURL=decodeURL so the decoded url is returned.

Output:
 Decoded URL: https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs="vp8.0, vorbis"&quality=medium  
 Encoded URL2: https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%2Fmp4%26pl%3D21%26itag%3D22%26%26itag%3D43%26type%3Dvideo%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22%26quality%3Dmedium  

Full Program:
 package com.cooltrickshome;  
 import java.io.UnsupportedEncodingException;  
 import java.net.URLDecoder;  
 import java.net.URLEncoder;  
 public class URLEncodeDecode {  
      public static void main(String[] args) {  
           // TODO Auto-generated method stub  
           String url="https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%252Fmp4%26pl%3D21%26itag%3D22%26\u0026itag=43\u0026type=video%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22\u0026quality=medium";  
           String url2="https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs=\"vp8.0, vorbis\"&quality=medium";  
           String decodeURL = decode(url);  
           System.out.println("Decoded URL: "+decodeURL);  
           String encodeURL = encode(url2);  
           System.out.println("Encoded URL2: "+encodeURL);  
      }  
      public static String decode(String url)  
      {  
                try {  
                     String prevURL="";  
                     String decodeURL=url;  
                     while(!prevURL.equals(decodeURL))  
                     {  
                          prevURL=decodeURL;  
                          decodeURL=URLDecoder.decode( decodeURL, "UTF-8" );  
                     }  
                     return decodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while decoding" +e.getMessage();  
                }  
      }  
      public static String encode(String url)  
      {  
                try {  
                     String encodeURL=URLEncoder.encode( url, "UTF-8" );  
                     return encodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while encoding" +e.getMessage();  
                }  
      }  
 }  

Hope it helps :)

Wednesday, December 28, 2016

Create a Youtube metadata crawler using Java

This blog post will extract metadata information from youtube using Java. It makes use of JSoup library.

Youtube Video Metadata Crawled:
1) Comments
2) Likes/Dislikes
3) Number of user subscribed
4) Video Title
5) Number of views for video
6) Video Description

Language Used:
Java

POM Dependency:
  <dependency>  
       <groupId>org.jsoup</groupId>  
       <artifactId>jsoup</artifactId>  
       <version>1.8.3</version>  
  </dependency>  

Git Repo:
https://github.com/csanuragjain/extra/tree/master/YoutubeMetadataExtractor

Program:

main Method:
      public static void main(String[] args) {  
           String link="";  
           Scanner s=new Scanner(System.in);  
           // TODO Auto-generated method stub  
           try {  
                System.out.println("Enter the youtube video for which metadata need to be extracted");  
                link=s.nextLine();  
                Document doc = Jsoup.connect(link).ignoreContentType(true).timeout(5000).get();  
                YoutubeMetadataCrawler ymc=new YoutubeMetadataCrawler();  
                String title=ymc.getTitle(doc);  
                String desc=ymc.getDesc(doc);  
                String views=ymc.getViews(doc);  
                String subscribed=ymc.getPeopleSubscribed(doc);  
                int liked=ymc.getPeopleLiked(doc);  
                int disliked=ymc.getPeopleDisliked(doc);  
                String vid=ymc.getVideoId(link);  
                List<String> comments=ymc.getCommentsDesc(link,vid);  
                System.out.println(title);  
                System.out.println(desc);  
                System.out.println("Video Views: \n"+views);  
                System.out.println("People subscribed: \n"+subscribed);  
                System.out.println("People who liked the video: \n"+liked);  
                System.out.println("People who disliked the video: \n"+disliked);  
                System.out.println("Top Comments: ");  
                int i=0;  
                for(String comment:comments)  
                {  
                     System.out.println(++i+") "+comment);  
                }  
           } catch (IOException e) {  
                System.out.println("JSoup is unable to connect to the website");  
           }  
           finally  
           {  
                s.close();  
           }  
      }  

How it works:
1) We use the scanner object to obtain the youtube video url from user
2) We use JSOUP to connect to youtube video
3) We call the several modules for obtaining title, comments, title, description, likes etc and print the same to user.

removeUTFCharacters method:
      public static String removeUTFCharacters(String data){  
           Pattern p = Pattern.compile("\\\\u(\\p{XDigit}{4})");  
           Matcher m = p.matcher(data);  
           StringBuffer buf = new StringBuffer(data.length());  
           while (m.find()) {  
           String ch = String.valueOf((char) Integer.parseInt(m.group(1), 16));  
           m.appendReplacement(buf, Matcher.quoteReplacement(ch));  
           }  
           m.appendTail(buf);  
           return new String(buf);  
           }  

How it works:
1) This module removes the UTF characters.
2) It tries to find if any unicode character is present in the string passed.
3) If it finds any unicode character then it will replace it will its corresponding string

getCommentToken method:
      public String getCommentToken(Response res)  
      {  
           String pageSource=res.body();  
           String commentToken=pageSource=pageSource.substring(pageSource.indexOf("COMMENTS_TOKEN': \"")+18);  
           commentToken=commentToken.substring(0,commentToken.indexOf("\""));  
           commentToken=commentToken.replaceAll("%", "%25");  
           return commentToken;  
      }  

How it works:
1) If you open any youtube video, you will find below token in page source which is used by youtube for extracting comments.
 'COMMENTS_TOKEN': "SOME_TOKEN",  
2) We obtain a response object as argument. This contains the response details when user provided youtube video is opened.
3)  In our module, first we obtain the page source of the youtube video page by calling body method over the response object obtained as argument.
4) Now we search for the text COMMENTS_TOKEN': " in the page source and extract the string upto ". This will extract the comment token.
5) Comment token is returned.


getSessionToken method:
 public String getSessionToken(Response res)  
      {  
           String pageSource=res.body();  
           String xsrfToken=pageSource=pageSource.substring(pageSource.indexOf("XSRF_TOKEN': \"")+14);  
           xsrfToken=xsrfToken.substring(0,xsrfToken.indexOf("\""));  
           return xsrfToken;  
      }  

How it works:
1) If you open any youtube video, you will find below token in page source which is used by youtube for csrf protection.
 'XSRF_TOKEN':"SOME_TOKEN",  
2) We obtain a response object as argument. This contains the response details when user provided youtube video is opened.
3)  In our module, first we obtain the page source of the youtube video page by calling body method over the response object obtained as argument.
4) Now we search for the text 'XSRF_TOKEN':" in the page source and extract the string upto ". This will extract the xsrf token.
5) XSRF token is returned.

getVideoId method:
 public String getVideoId(String url)  
      {  
           url=url.substring(url.indexOf("v=")+2);  
           if(url.contains("?"))  
           {  
                url=url.substring(0,url.indexOf("?"));  
           }  
           return url;  
      }  

How it works:
1) Youtube url contains a param named v which contains the video id
2) We find the location of word v= and extract its value and return the same.

getTitle method:
 public String getTitle(Document doc)  
      {  
           return doc.select("#eow-title").text();  
      }  

How it works:
1) Youtube video contains the video title in HTML element with id as eow-title
 <span id="eow-title" class="watch-title" dir="ltr" title="Dangal | Official Trailer | Aamir Khan | In Cinemas Dec 23, 2016">  
   Dangal | Official Trailer | Aamir Khan | In Cinemas Dec 23, 2016  
  </span>  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing id as eow-title
4) We return the title

getViews method:
 public String getViews(Document doc)  
      {  
           return doc.select(".watch-view-count").text();  
      }  

How it works:
1) Youtube video contains the video views in HTML element with class as watch-view-count
 <div class="watch-view-count">40,095,041 views</div>  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing class as watch-view-count
4) We return the views

getDesc method:
 public String getDesc(Document doc)  
      {  
           return doc.select("#watch-description-clip").text();  
      }  

How it works:
1) Youtube video contains the video description in HTML element with id as watch-description-clip
 <div id="watch-description-clip"><div id="watch-uploader-info"><strong class="watch-time-text">Some description  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing id as watch-description-clip
4) We return the video description

getPeopleSubscribed method:
 public int getPeopleSubscribed(Document doc)  
      {  
           return doc.select(".yt-subscriber-count").text().replace(",", "");  
      }  

How it works:
1) Youtube video contains the user subscribed in HTML element with class as yt-subscriber-count
 <span class="yt-subscription-button-subscriber-count-branded-horizontal yt-subscriber-count" title="1,057,873" aria-label="1,057,873" tabindex="0">1,057,873</span>  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing class as yt-subscriber-count
4) We return the video subscription count

getPeopleLiked method:
 public int getPeopleLiked(Document doc)  
      {  
           return Integer.parseInt(doc.select("button.like-button-renderer-like-button-unclicked span").text().replace(",", ""));  
      }  

How it works:
1) Youtube video contains the people who liked video in HTML element with class as like-button-renderer-like-button-unclicked
 <button class="yt-uix-button yt-uix-button-size-default yt-uix-button-opacity yt-uix-button-has-icon no-icon-markup like-button-renderer-like-button like-button-renderer-like-button-unclicked yt-uix-clickcard-target  yt-uix-tooltip" type="button" onclick=";return false;" title="I like this" aria-label="like this video along with 327,229 other people" data-force-position="true" data-orientation="vertical" data-position="bottomright" data-tooltip-text="I like this" aria-labelledby="yt-uix-tooltip620-arialabel"><span class="yt-uix-button-content">327,229</span></button>  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing class as like-button-renderer-like-button-unclicked
4) We return the video like count

getPeopleDisliked method:
 public int getPeopleDisliked(Document doc)  
      {  
           return Integer.parseInt(doc.select("button.like-button-renderer-dislike-button-unclicked span").text().replace(",", ""));  
      }  

How it works:
1) Youtube video contains the people who disliked video in HTML element with class as like-button-renderer-dislike-button-unclicked
 <button class="yt-uix-button yt-uix-button-size-default yt-uix-button-opacity yt-uix-button-has-icon no-icon-markup like-button-renderer-dislike-button like-button-renderer-dislike-button-unclicked yt-uix-clickcard-target  yt-uix-tooltip" type="button" onclick=";return false;" title="I dislike this" aria-label="dislike this video along with 28,697 other people" data-force-position="true" data-orientation="vertical" data-position="bottomright" data-tooltip-text="I dislike this" aria-labelledby="yt-uix-tooltip621-arialabel"><span class="yt-uix-button-content">28,697</span></button>  
2) Document object contains the document detail of the youtube video.
3) We use the document object to find the text from the HTML tag containing class as like-button-renderer-dislike-button-unclicked
4) We return the video dislike count

Output:
 Enter the youtube video for which metadata need to be extracted  
 https://www.youtube.com/watch?v=x_7YlGv9u1g  
 Dangal | Official Trailer | Aamir Khan | In Cinemas Dec 23, 2016  
 Published on Oct 19, 2016 Dangal is an extraordinary true story based on the life of Mahavir Singh and his two daughters, Geeta and Babita Phogat. The film traces the inspirational journey of a father who trains his daughters to become world class wrestlers. Release Date: 23rd December 2016 Starring: Aamir Khan, Sakshi Tanwar, Fatima Sana Shaikh, Sanya Malhotra, Zaira Wasim, Suhani Bhatnagar Directed By: Nitesh Tiwari Written By: Nitesh Tiwari, Shreyas Jain, Piyush Gupta, Nikhil Mehrotra Produced By: Aamir Khan, Kiran Rao & Siddharth Roy Kapur Music: Pritam Lyrics: Amitabh Bhattacharya Director of Photography: Setu Production Designer: Laxmi Keluskar & Sandeep Meher Editor: Ballu Saluja Casting Director: Mukesh Chhabra Costume Designer: Maxima Basu Wrestling Choreography and Coach: Kripashankar Patel Bishnoi Action Director: Sham Kaushal Sound Designer: Shajith Koyeri SUBSCRIBE UTV Motion Pictures: http://www.youtube.com/utvmotionpictures Keep up with UTV Motion Pictures on: TWITTER: https://twitter.com/utvfilms FACEBOOK: https://www.facebook.com/utvmotionpic... INSTAGRAM: http://instagram.com/utvfilms GOOGLE+: https://plus.google.com/+UTVMotionPic... PINTEREST: http://pinterest.com/utvfilms/ Category Entertainment License Standard YouTube License  
 Video Views:   
 42,718,537 views  
 People subscribed:   
 1072189  
 People who liked the video:   
 333530  
 People who disliked the video:   
 29338  
 Top Comments:   
 1) Where are the haters who wanted to boycott this movie. The movie has already crossed 100 crores in 3 days.  
 2) No Words To Express The AWESOMENESS Of This Movie...😘😍😎😯  
 3) AWESOME MOVIE BEST MOVIE OF 2016 YALL WATCH IT  

Full Program:
 package com.cooltrickshome;  
 import java.io.IOException;  
 import java.util.ArrayList;  
 import java.util.List;  
 import java.util.Scanner;  
 import java.util.regex.Matcher;  
 import java.util.regex.Pattern;  
 import org.jsoup.Connection.Response;  
 import org.jsoup.Jsoup;  
 import org.jsoup.nodes.Document;  
 import org.jsoup.nodes.Element;  
 import org.jsoup.select.Elements;  
 public class YoutubeMetadataCrawler {  
      /**  
       * @param args  
       */  
      public static void main(String[] args) {  
           String link="";  
           Scanner s=new Scanner(System.in);  
           // TODO Auto-generated method stub  
           try {  
                System.out.println("Enter the youtube video for which metadata need to be extracted");  
                link=s.nextLine();  
                Document doc = Jsoup.connect(link).ignoreContentType(true).timeout(5000).get();  
                YoutubeMetadataCrawler ymc=new YoutubeMetadataCrawler();  
                String title=ymc.getTitle(doc);  
                String desc=ymc.getDesc(doc);  
                String views=ymc.getViews(doc);  
                String subscribed=ymc.getPeopleSubscribed(doc);  
                int liked=ymc.getPeopleLiked(doc);  
                int disliked=ymc.getPeopleDisliked(doc);  
                String vid=ymc.getVideoId(link);  
                List<String> comments=ymc.getCommentsDesc(link,vid);  
                System.out.println(title);  
                System.out.println(desc);  
                System.out.println("Video Views: \n"+views);  
                System.out.println("People subscribed: \n"+subscribed);  
                System.out.println("People who liked the video: \n"+liked);  
                System.out.println("People who disliked the video: \n"+disliked);  
                System.out.println("Top Comments: ");  
                int i=0;  
                for(String comment:comments)  
                {  
                     System.out.println(++i+") "+comment);  
                }  
           } catch (IOException e) {  
                System.out.println("JSoup is unable to connect to the website");  
           }  
           finally  
           {  
                s.close();  
           }  
      }  
      public static String removeUTFCharacters(String data){  
           Pattern p = Pattern.compile("\\\\u(\\p{XDigit}{4})");  
           Matcher m = p.matcher(data);  
           StringBuffer buf = new StringBuffer(data.length());  
           while (m.find()) {  
           String ch = String.valueOf((char) Integer.parseInt(m.group(1), 16));  
           m.appendReplacement(buf, Matcher.quoteReplacement(ch));  
           }  
           m.appendTail(buf);  
           return new String(buf);  
           }  
      public String getTitle(Document doc)  
      {  
           return doc.select("#eow-title").text();  
      }  
      public String getViews(Document doc)  
      {  
           return doc.select(".watch-view-count").text();  
      }  
      public String getDesc(Document doc)  
      {  
           return doc.select("#watch-description-clip").text();  
      }  
      public List<String> getCommentsDesc(String link, String vid)  
      {  
           List<String> comments=new ArrayList<>();  
           try {  
                Response response = Jsoup.connect(link).ignoreContentType(true).timeout(5000).execute();  
                Document doc = Jsoup.connect("https://www.youtube.com/watch_fragments_ajax?v="+vid+"&tr=time&distiller=1&ctoken="+getCommentToken(response)+"&frags=comments&spf=load")  
                     .ignoreContentType(true)  
                     .cookies(response.cookies())  
                     .header("X-Client-Data", "")  
                     .timeout(5000)  
                     .data("session_token",getSessionToken(response))  
                     .post();  
                String commentSource=doc.body().text();  
                while(commentSource.indexOf("comment-renderer-text-content")>-1)  
                {  
                     int pos=commentSource.indexOf("comment-renderer-text-content")+37;  
                     commentSource=commentSource.substring(pos);  
                     //System.out.println(commentSource);  
                     String comment=commentSource.substring(0,commentSource.indexOf("div")-8);  
                     comments.add(removeUTFCharacters(comment));  
                }  
           Elements e= doc.select(".comment-renderer-text-content");  
           for(Element e1:e)  
           {  
                comments.add(e1.text());  
           }  
           } catch (IOException e2) {  
                System.out.println("Unable to retrieve comments "+e2.getMessage());  
                e2.printStackTrace();  
           }  
           return comments;  
      }  
      public String getCommentToken(Response res)  
      {  
           String pageSource=res.body();  
           String commentToken=pageSource=pageSource.substring(pageSource.indexOf("COMMENTS_TOKEN': \"")+18);  
           commentToken=commentToken.substring(0,commentToken.indexOf("\""));  
           commentToken=commentToken.replaceAll("%", "%25");  
           return commentToken;  
      }  
      public String getSessionToken(Response res)  
      {  
           String pageSource=res.body();  
           String xsrfToken=pageSource=pageSource.substring(pageSource.indexOf("XSRF_TOKEN': \"")+14);  
           xsrfToken=xsrfToken.substring(0,xsrfToken.indexOf("\""));  
           return xsrfToken;  
      }  
      public String getVideoId(String url)  
      {  
           url=url.substring(url.indexOf("v=")+2);  
           if(url.contains("?"))  
           {  
                url=url.substring(0,url.indexOf("?"));  
           }  
           return url;  
      }  
      public int getPeopleSubscribed(Document doc)  
      {  
           return doc.select(".yt-subscriber-count").text().replace(",", "");  
      }  
      public int getPeopleLiked(Document doc)  
      {  
           return Integer.parseInt(doc.select("button.like-button-renderer-like-button-unclicked span").text().replace(",", ""));  
      }  
      public int getPeopleDisliked(Document doc)  
      {  
           return Integer.parseInt(doc.select("button.like-button-renderer-dislike-button-unclicked span").text().replace(",", ""));  
      }  
 }  


Hope it helps :)

Friday, November 4, 2016

Creating a Subdomain Crawler using Java

Running the Subdomain Crawler:

1) Copy the git project from https://github.com/csanuragjain/subdomainCrawler

2) Create a Maven Project in Eclipse. Add the content from the above git project to your eclipse project

3) Run Driver.java

4) Enter the domain for which you would like to extract subdomains

5) Subdomain extraction would start

6) You can extend the capability by adding more search engines.

7) You can export the project from Eclipse so that it can work as standalone.

Search engines Used for Crawling:

Google, Ask, Baidu, Bing, Crtsh, DNSDumpster, Netcraft, Passive DNS, Virustotal, SSLSAN, ThreatCrowd, Yahoo


Project Explanation:

1) Driver folder contains the chromedriver which is used by selenium

2) Crawler folder contains extraction rules for each Search engine

3) Driver is the main class

4) subdomainList variable in Driver class will hold all the subdomains found

Let me know if you have any doubt on any of the program step or you could suggest some new search engines which we can add to the list.