URL Encoding and Decoding using Java

It is common requirement to implement URL encoding and decoding in Java while creating crawlers or downloaders. This post focus on creating modules for encoding and decoding of the passed url using Java.

Language Used:
Java

Git Location:
https://github.com/csanuragjain/extra/tree/master/EncodeDecodeURL

Program:
main method:

 public static void main(String[] args) {  
           // TODO Auto-generated method stub  
           String url="https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%252Fmp4%26pl%3D21%26itag%3D22%26\u0026itag=43\u0026type=video%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22\u0026quality=medium";  
           String url2="https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs=\"vp8.0, vorbis\"&quality=medium";  
           String decodeURL = decode(url);  
           System.out.println("Decoded URL: "+decodeURL);  
           String encodeURL = encode(url2);  
           System.out.println("Encoded URL2: "+encodeURL);  
      }  

How it works:
1) url is the variable having encoded url which we want to decode
2) url2 is the variable having an url which we want to encode
3) We call the decode method which decodes the url and then print the same.
4) We call the encode method which encodes the url2 and then print the same.

encode method:

      public static String encode(String url)  
      {  
                try {  
                     String encodeURL=URLEncoder.encode( url, "UTF-8" );  
                     return encodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while encoding" +e.getMessage();  
                }  
      }  

How it works:
1) We use the encode method of a predefined java class named URLEncoder
2) encode method of URLEncoder takes 2 arguments
3) 1st argument defines the url to be encoded
4) 2nd argument defines the encoding scheme to be used
5) After encoding the resulting encoded url is returned

decode method:

      public static String decode(String url)  
      {  
                try {  
                     String prevURL="";  
                     String decodeURL=url;  
                     while(!prevURL.equals(decodeURL))  
                     {  
                          prevURL=decodeURL;  
                          decodeURL=URLDecoder.decode( decodeURL, "UTF-8" );  
                     }  
                     return decodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while decoding" +e.getMessage();  
                }  
      }  

How it works:
1) Since same url can be encoded multiple times so we need to decode until url cannnot be decoded further.
2) For eg: "video%252Fmp4" is result of 2 encoding. On decoding it once we get "video%2Fmp4". Now url need to be further decoded so when we apply decoding again we get "video/mp4", which is the result.
3) We use the decode method of a predefined java class named URLDecoder
4) decode method of URLDecoder takes 2 arguments
5) 1st argument defines the url to be decoded
6) 2nd argument defines the decoding scheme to be used
7) After decoding the resulting decoded url is returned
8) We create 2 variables prevURL which is empty and decodeURL which contain the url to be decoded.

 Variable State:  
 prevURL = ""  
 decodeURL ="somethingvideo%252Fmp4"  

9) We create an iteration which runs until prevURL!=decodeURL
10) Now we update prevURL to decodeURL and update decodeURL with the decoded value of the url passed.

 Variable State:  
 prevURL = "somethingvideo%252Fmp4"  
 decodeURL ="somethingvideo%2Fmp4"  

11) Since prevURL!=decodeURL so Step 10 runs again

 Variable State:  
 prevURL = "somethingvideo%2Fmp4"  
 decodeURL ="somethingvideo/mp4"  

12) Since prevURL!=decodeURL so Step 10 runs again

 Variable State:  
 prevURL = "somethingvideo/mp4"  
 decodeURL ="somethingvideo/mp4"  

13) Since prevURL=decodeURL so the decoded url is returned.

Output:

 Decoded URL: https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs="vp8.0, vorbis"&quality=medium  
 Encoded URL2: https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%2Fmp4%26pl%3D21%26itag%3D22%26%26itag%3D43%26type%3Dvideo%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22%26quality%3Dmedium  

Full Program:

 package com.cooltrickshome;  
 import java.io.UnsupportedEncodingException;  
 import java.net.URLDecoder;  
 import java.net.URLEncoder;  
 public class URLEncodeDecode {  
      public static void main(String[] args) {  
           // TODO Auto-generated method stub  
           String url="https%3A%2F%2Fr1---sn-ci5gup-cags.googlevideo.com%2Fvideoplayback%3Fpcm2cms%3Dyes%26mime%3Dvideo%252Fmp4%26pl%3D21%26itag%3D22%26\u0026itag=43\u0026type=video%2Fwebm%3B+codecs%3D%22vp8.0%2C+vorbis%22\u0026quality=medium";  
           String url2="https://r1---sn-ci5gup-cags.googlevideo.com/videoplayback?pcm2cms=yes&mime=video/mp4&pl=21&itag=22&&itag=43&type=video/webm; codecs=\"vp8.0, vorbis\"&quality=medium";  
           String decodeURL = decode(url);  
           System.out.println("Decoded URL: "+decodeURL);  
           String encodeURL = encode(url2);  
           System.out.println("Encoded URL2: "+encodeURL);  
      }  
      public static String decode(String url)  
      {  
                try {  
                     String prevURL="";  
                     String decodeURL=url;  
                     while(!prevURL.equals(decodeURL))  
                     {  
                          prevURL=decodeURL;  
                          decodeURL=URLDecoder.decode( decodeURL, "UTF-8" );  
                     }  
                     return decodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while decoding" +e.getMessage();  
                }  
      }  
      public static String encode(String url)  
      {  
                try {  
                     String encodeURL=URLEncoder.encode( url, "UTF-8" );  
                     return encodeURL;  
                } catch (UnsupportedEncodingException e) {  
                     return "Issue while encoding" +e.getMessage();  
                }  
      }  
 }  

Hope it helps :)

Create a File Downloader with Progress status in Java

Overview:
1) It is very easy to download files with Java using the URL class.
2) Download files gets important while creating Automation Job. For eg. if you need to download the daily builds and then perform your test over them then you can use this.

Where to Download:
Please download the full code from: https://github.com/csanuragjain/filedownloader & then run FileDownloader.java

Program Explanation:

Main method:

      static int downloadinProgress = 0;  
      static File f = new File("");  
      public static void main(String[] args) throws MalformedURLException,  
                IOException {  
           Scanner s = new Scanner(System.in);  
           String ch = "YES";  
           while (!ch.equals("NO")) {  
                System.out.println("Enter the url to download from");  
                final String downloadURL = s.nextLine();  
                new Thread() {  
                     public void run() {  
                          new FileDownloader().downloadFile(downloadURL);  
                     }  
                }.start();  
                System.out.println("Do you wish to download more files (YES/NO)");  
                ch = s.nextLine();  
           }  
      }  

How it works:

1) We created 2 static variables, downloadinProgress - tells how many active downloads are their & f - Used later to get the current File path where downloaded files are kept
2) Scanner s is used for taking user input
3) We ask user for the download url and pass the same as argument inside downloadFile function
4) downloadFile function is called within anonymous thread so that we can have multiple file downloads
5) Step 3 keeps on repeating until user press NO

downloadFile method:

      public void downloadFile(String downloadURL) {  
           downloadinProgress++;  
           System.out.println("**Download stats: Number of ongoing downloads: "  
                     + downloadinProgress);  
           BufferedInputStream in = null;  
           RandomAccessFile fout = null;  
           String fileName = "";  
           long fileSize = 0;  
           long downloaded = 0;  
           try {  
                URL u = new URL(downloadURL);  
                URLConnection uc = u.openConnection();  
                uc.setRequestProperty("Range", "bytes=0-");  
                fileName = u.getFile();  
                fileName = fileName.substring(fileName.lastIndexOf('/') + 1);  
                fileSize = uc.getContentLength();  
                System.out.println("**Download stats: Downloading " + fileName  
                          + " which has size of " + (fileSize / 1000) + "KB");  
                in = new BufferedInputStream(uc.getInputStream());  
                fout = new RandomAccessFile(fileName, "rw");  
                fout.seek(downloaded);  
                final byte data[] = new byte[1024];  
                int count;  
                boolean ALERT_WHEN_50_PERCENT_COMPLETE = true;  
                while ((count = in.read(data, 0, 1024)) != -1) {  
                     fout.write(data, 0, count);  
                     downloaded += count;  
                     if (ALERT_WHEN_50_PERCENT_COMPLETE  
                               && downloaded * 2 >= fileSize) {  
                          System.out.println("**Download stats: " + fileName  
                                    + ": Completed 50% download...");  
                          ALERT_WHEN_50_PERCENT_COMPLETE = false;  
                     }  
                }  
                System.out  
                          .println("**Download stats: Download completed. File at: "  
                                    + f.getAbsolutePath() + File.separator + fileName);  
                downloadinProgress--;  
           } catch (Exception e) {  
                System.out.println("**Download stats: Download Failed for "  
                          + fileName);  
                downloadinProgress--;  
           } finally {  
                if (in != null) {  
                     try {  
                          in.close();  
                     } catch (IOException e) {  
                          // TODO Auto-generated catch block  
                          e.printStackTrace();  
                     }  
                }  
                if (fout != null) {  
                     try {  
                          fout.close();  
                     } catch (IOException e) {  
                          // TODO Auto-generated catch block  
                          e.printStackTrace();  
                     }  
                }  
           }  
      }  

How it works:

1) Increment downloadinProgress since a new download has begun
2) Now we open a connection to the download url
3) We pass range as bytes=0- which tells that i am interested in downloading the content from byte 0 to last byte. This was not neccessary here but it is very useful when you are creating a downloader with resume capacity. So assume if download failed at 100 byte then you can restart your download with range as 100-
4) Now we extract the filename using getFile and filesize using getContentLength
5) We define a RandomAccessFile object called fout and point this to the last byte on the file where we are going to write the downloaded content. In our case it will point to starting of file since our download file did not exist. RandomAccessFile is useful when you want to resume a download and want to write in file from the point where download failed.
6) Now we get the input stream from url connection and start writing the content to the resulting file at 1024 bytes per iteration
7) We kept a tracker ALERT_WHEN_50_PERCENT_COMPLETE which will notify when 50% of file download is complete using the downloaded variable which tracks the number of bytes written to file. You can use the same trick to update a progress bar when using swing
8) After this completes, objects are closed
9) File download is complete

Hope it helps :)