Wednesday, July 16, 2014

Don't return in a closure for each() in Groovy

I am a Java coder.  All these years I heard so many buzz about functional language, but I never took the dive.  Well, I got some tastes recently. 

We recently switched to use Gradle to manage the project.  I was banning my head for a couple hours yesterday for an odd problem.


boolean isStartedWithList(String str){
  def list=['pre','post']
  list.each {
    if (str.startsWith(it)){
       println 'hit true'
       return true
    }
   }
  println 'hit default false'
  return false
  }  
If you run isStartedWithList('prehistorical'), you will see
hit true
hit default false
The returned value is false. This is because the (first and the second) return only gets you out of the closure, not the method. There are two ways to fix it: to switch back to the traditional for loop, or not to rely on the early exit from the return call.
boolean isStartedWithList(String str){
  def list=['pre','post']
  for (it:list) {
    if (str.startsWith(it)){
       println 'hit true'
       return true
    }
  println 'hit false'
  return false
  }  
boolean isStartedWithList(String str){
  def list=['pre','post']
  result=false;
  list.each {
    result=result||str.startsWtih(it)
   }
  return result
  }  
Both will get you the expected answer. Lessons are learned!

Edit 7/18/2014:

I just learned a even better one using the groovy collection method any
boolean isStartedWithList(String str){
  return ['pre','post'].any{str.startsWith(it)}
}

Sunday, June 3, 2012

A combiner works in a reducer as well as a mapper

In Hadoop, a combiner is usually introduced as a way to collapse the output record of maps in the end of a map task and therefore reduce the network traffic and alleviate the load of reducers. A less known function of a combiner that I recently discovered is that  it also works in the reduce phase. (CDH3u3, 0.20.2).  A reducer task, more specifically, its shuffle functions,  usually kicks off before all maps finish.  Once it collects enough records from all maps tasks (controlled by "mapred.inmem.merge.threshold" property) through network, records will be merged and a combiner will be called upon those records if one is specified in the job conf.  This is a very useful feature in the case where there are too many map tasks for the system to start them simultaneously.  Map outputs are collapsed by the combiner in the reducer task during the time, and final reduce run will not need to work no more records than "mapred.inmem.merge.threshold".  Many commercial or third-part versions of Hadoop actually specifically advertise some function similar as "proactive execution" or so.  It is great to know that the plain vanilla Hadoop already has something like this.

Saturday, November 12, 2011

Using a csv file as a metadata database in Greenstone (2.84)

Alternative to entering metadate manually in GLI in Greenstone (currently 2.84).  One can create a csv file with all the metadatas and import it into greenstone using gli.  This is very useful for at least two cases: 
  1. You can get the metadata from somewhere else in csv format.
  2. You want to experiment on a collection.  And you manage metadata in a separate csv file and  do not need to reenter all the metadatas after you screw up the collection. 

Greenstone has a metadatacsv plugin and unfortunately the author cannot figure out how to use it.  Alternatively, Greenstone has a explode metadata mechanism which is very useful.   Let us detail it. 

Let us first assume we have three files in Greenstone:

a.txt, b.txt, c.html


Filename,dc.Title,dc.Creator,Description,Contributor
a.txt, a, GZ, first test, GJZ
b.txt, b, GZ, second test, GJZ
c.html, c, GZ, third one, GJZ

A couple of point to notice for this file:
  • The first line is the label
  • The name of the label must exactly match the your metadata name (including the case, title will be mapped into "dc.title", instead of the standard "dc.Title")
  • Later we will choose the default metadata set, the non-qualified label will be mapped into this default set (Dublin Core in this case)
  • One can use a simple text editor do it. 
  • Google Doc is also an excellent tool with the  built-in collaborating function. One can create a spreadsheet first, fill it with your team mates, then "download as" a csv file. 

Here is the procedure in GLI
  1. (optional) create a new collection
  2. In the "Gather" tab, add all three files (a.txt, b.txt, c.html)
  3. In the "Gather" tab, add meta.csv (a window will pops up asking about plugin to use, choose either one and this does not matter much)
  4. In the "Enrich" tab, right-click "meta.csv", choose "explode metadata database", a new window will pop up (it shows as CSVPlugin, it is fine)
  5. choose "metadata_set" accordingly (Dublin Core in the example), this will be your default set, all your non-qualified metadata will be mapped into this (in our example, "Description" is mapped as "dc.Description")
  6. tick "document_field", enter the label of the file name column ("Filename" in the example)
  7. click "explode"
  8. In the Enrich tab, you will notice that all three files becomes the sub-levels of meta. And the metadata field is populated.  You can enter more
  9. Build the collection, tweak the display, do the rest 


Adding metadata with a new metadata scheme


Sometimes one has some special metadata fields that do not fit any existing schema.  One might want to create a new one from scratch or by modifying the existing one.  This can be done in Greenstone GEMS. Or one can do the same in  GLI "Manage Metadatasets..." in "Enrich" tab. 



I made a new metadata set "my new DC" from Dublin Core.  So it has no complication of the sublevel, all flat.  I specify it with a new "namespace"  ---"mdc" in order to distinguish it from normal dc.  I then use the same procedure as in the coruse website to explode a csv metadata file and it work fine. 



Subtlety with line end in CSV

Sometimes it just gives you a 000001.nul when you explode the csv file. It seems that CSVplugin of greenstone (@2.84) only works well with unix time line end "LR", but not well with Windows type "CR LR". Unfortunately, Excel generates wrong style. One way around is to copy the full sheet from excel to google doc spreadsheet and then "download as" CSV. Google doc seems to produce the CSV Greenstone likes. This csv file should explode fine. Greenstone team needs to work on the CSVplugin.

Thursday, November 10, 2011

Adding Media Player into Greenstone with jPlayer


"jPlayer" is a very nice javascript libraryto play various types of media files in browser using html 5 or flash. It can be integrated into the digital library software Greenstone (2.84) to enable it play various digital files. In this example, we enable the following types:
audio: mp3, m4a, oga
video: mp4(m4v), webmv, ogv, flv
(note: These are not all playable in every browsers due to the inconsistent media-type support across them.)

Let us assume the collection folder is {mycollection}="{greenstone folder}/collect/mycollection"

This example uses jPlayer 2.1.0, Greenstone 2.84.




  1.  Download jPlayer files and expand them into the "{mycollection}/script" folder
    ./add-on/jplayer.playlist.min.js
    ./add-on/jquery.jplayer.inspector.js
    ./Jplayer.swf
    ./jquery.jplayer.min.js
    
  2. Download one of the jPlayer skins (Blue Monday : Pink Flag) and expand all files except the *.psd file into the "{mycollection}/style" folder
    jplayer.pink.flag.css
    jplayer.pink.flag.jpg
    jplayer.pink.flag.seeking.gif
    jplayer.pink.flag.video.play.png
    
  3. Put a new player button image into "{mycollection}/images" folder.  I use this link. A small size one (48px or even 16px) is enough. One can even skip this step, if a "player" button or pure text is preferred.
  4. Create a new javascript file (playMedia.js) with the following text and put it into "{myCollection}/script" folder
    function playMedia(playerId,inspectorId,fullcontainerId,containerId,base,type,src){
    
     $("#"+fullcontainerId).show();
     $("#"+playerId).jPlayer("destroy");
          $("#"+playerId).jPlayer({
            ready: function () {
              $(this).jPlayer("pauseOthers").jPlayer("setMedia",media(type,base+"/"+src)).jPlayer("play");
            },
            errorAlerts:false,
            warnAlerts:false,
            swfPath: base+"/script/",
            supplied: type,
            cssSelectorAncestor: "#"+containerId,
            solution:"html,flash",
            ended:function(){
                
             $("#"+fullcontainerId).hide();
            }
          });
          $("#"+inspectorId).jPlayerInspector({jPlayer:$("#"+playerId)});
    }
    
    function media(type,src){
    
    switch (type.toLowerCase()){
    case "mp3":return {mp3:src}; break;
    case "oga":return {oga:src}; break;
    case "ogv":return {ogv:src}; break;
    case "m4a":return {m4a:src}; break;
    case "webmv":return {webmv:src}; break;
    case "m4v":return {m4v:src}; break;
    case "ogv":return {ogv:src}; break;
    case "flv":return {flv:src}; break;
    default:
    }
    }
    
    
  5. Start GLI of greenstone, open collection, paste the following into the "Collection Specific Macros" in "Format" tab. Alternatively, one can create a file "extra.dm" in "{myCollection}/macros" folder
    
    
    # extra.dm file
    # You can add collection specific macros in here
    # Lines starting with a '#' are comments.
    # Remember to include the package declaration
    
    package Style
    
    # will be applied to all pages 
    # add css style lines inside the style tags
    _collectionspecificstyle_ {
    <style type="text/css">
    div.jp-audio,
    div.jp-video {
      /* Edit the font-size to counteract inherited font sizing.
       * Eg. 1.25em = 1 / 0.8em
       */
      font-size:1.25em;
    \}
    </style>
    }
    
    # add any javascript functions here
    _collectionspecificscript_ { 
    $(document).ready(function(){
          $("#jquery-jplayer-1").jPlayer({
            ready: function () {
              
            \},
            errorAlerts:false,
            swfPath: "_httpcollection_/script/",
            supplied: "mp3,oga,m4a",
            cssSelectorAncestor: '#jp-container-1',
            solution:"html,flash",
            ended:function(){
            $("#jplayer-div").hide();
            \}
          \});
        $("#jplayer-inspector").jPlayerInspector({jPlayer:$("#jquery-jplayer-1")\});
    
    
    $("#jquery-jplayer-video").jPlayer({
            ready: function () {
             
            \},
            errorAlerts:false,
            swfPath: "_httpcollection_/script/",
            supplied: "webmv,ogv,m4v",
            cssSelectorAncestor: '#jp-container-video',
            solution:"html,flash",
            ended:function(){
            $("#jplayer-div-video").hide();
            \}
          \});
        $("#jplayer-inspector-video").jPlayerInspector({jPlayer:$("#jquery-jplayer-video")\});
    
        \});
    
    function play(type,src){
    type=type.toLowerCase();
    switch (type.toLowerCase()){
    case "mp3":
    case "oga":
    case "m4a":$('#jplayer-div-video').hide();
       playMedia("jquery-jplayer-1","jplayer-inspector","jplayer-div","jp-container-1","_httpcollection_",type,src);
       break;
    case "webmv":
    case "m4v":
    case "flv":
    case "ogv":$('#jplayer-div').hide();
       playMedia("jquery-jplayer-video","jplayer-inspector-video","jplayer-div-video","jp-container-video","_httpcollection_",type,src);
       break;
    default:
    \}
    \}
    
    }
    
    
    _pagescriptfileextra_{
    <link href="_httpcollection_/style/jplayer.pink.flag.css" rel="stylesheet" type="text/css" />
    <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.6/jquery.min.js"></script>
    <script type="text/javascript" src="_httpcollection_/script/jquery.jplayer.min.js"></script>
    <script type="text/javascript" src="_httpcollection_/script/add-on/jquery.jplayer.inspector.js"></script>
    <script type="text/javascript" src="_httpcollection_/script/playMedia.js"></script>
    
    }
    
    
    
    _document:footer_ {
     _jplayer_
    </div> <!-- document:footer -->
    
    <div class="navarrowsbottom">
    _navarrowsbottom_
    </div>
    _endspacer__htmlfooter_
    }
    
    _jplayer_{
    
      <div id="jplayer-div" style="display:none;">
      <div id="jp-container-1" class="jp-audio" ">
    <div id="jquery-jplayer-1" class="jp-jplayer"></div>
      
    <div class="jp-type-single">
          <div class="jp-gui jp-interface">
            <ul class="jp-controls">
              <li><a href="javascript:;" class="jp-play" tabindex="1">play</a></li>
              <li><a href="javascript:;" class="jp-pause" tabindex="1">pause</a></li>
              <li><a href="javascript:;" class="jp-stop" tabindex="1">stop</a></li>
              <li><a href="javascript:;" class="jp-mute" tabindex="1" title="mute">mute</a></li>
              <li><a href="javascript:;" class="jp-unmute" tabindex="1" title="unmute">unmute</a></li>
              <li><a href="javascript:;" class="jp-volume-max" tabindex="1" title="max volume">max volume</a></li>
            </ul>
            <div class="jp-progress">
              <div class="jp-seek-bar">
                <div class="jp-play-bar"></div>
              </div>
            </div>
            <div class="jp-volume-bar">
              <div class="jp-volume-bar-value"></div>
            </div>
            <div class="jp-time-holder">
              <div class="jp-current-time"></div>
              <div class="jp-duration"></div>
              <ul class="jp-toggles">
                <li><a href="javascript:;" class="jp-repeat" tabindex="1" title="repeat">repeat</a></li>
                <li><a href="javascript:;" class="jp-repeat-off" tabindex="1" title="repeat off">repeat off</a></li>
              </ul>
            </div>
          </div>
          
          <div class="jp-no-solution">
            <span>Update Required</span>
            To play the media you will need to either update your browser to a recent version or update your <a href="http://get.adobe.com/flashplayer/" target="\_blank">Flash plugin</a>.
          </div>
        </div>
     
      </div>
      <div id="jplayer-inspector"></div>
    </div>
    
    <div id="jplayer-div-video" style="display:none;">
    <div id="jp-container-video" class="jp-video ">
        <div class="jp-type-single">
          <div id="jquery-jplayer-video" class="jp-jplayer"></div>
          <div class="jp-gui">
            <div class="jp-video-play">
              <a href="javascript:;" class="jp-video-play-icon" tabindex="1">play</a>
            </div>
            <div class="jp-interface">
              <div class="jp-progress">
                <div class="jp-seek-bar">
                  <div class="jp-play-bar"></div>
                </div>
              </div>
              <div class="jp-current-time"></div>
              <div class="jp-duration"></div>
              <div class="jp-controls-holder">
                <ul class="jp-controls">
                  <li><a href="javascript:;" class="jp-play" tabindex="1">play</a></li>
                  <li><a href="javascript:;" class="jp-pause" tabindex="1">pause</a></li>
                  <li><a href="javascript:;" class="jp-stop" tabindex="1">stop</a></li>
                  <li><a href="javascript:;" class="jp-mute" tabindex="1" title="mute">mute</a></li>
                  <li><a href="javascript:;" class="jp-unmute" tabindex="1" title="unmute">unmute</a></li>
                  <li><a href="javascript:;" class="jp-volume-max" tabindex="1" title="max volume">max volume</a></li>
                </ul>
                <div class="jp-volume-bar">
                  <div class="jp-volume-bar-value"></div>
                </div>
                <ul class="jp-toggles">
                  <li><a href="javascript:;" class="jp-full-screen" tabindex="1" title="full screen">full screen</a></li>
                  <li><a href="javascript:;" class="jp-restore-screen" tabindex="1" title="restore screen">restore screen</a></li>
                  <li><a href="javascript:;" class="jp-repeat" tabindex="1" title="repeat">repeat</a></li>
                  <li><a href="javascript:;" class="jp-repeat-off" tabindex="1" title="repeat off">repeat off</a></li>
                </ul>
              </div>
              
            </div>
          </div>
          <div class="jp-no-solution">
            <span>Update Required</span>
            To play the media you will need to either update your browser to a recent version or update your <a href="http://get.adobe.com/flashplayer/" target="\_blank">Flash plugin</a>.
          </div>
        </div>
        
      </div>
      <div id="jplayer-inspector-video"></div>
      </div>
    }
    
    
    
    
    A few notes for macro
    • "}" and "_" have to be escaped as "\_", "\}"
    • Two jPlayer instances and containers are created. One for audio and one for video.
    • A new macro "_jplayer_" is created.
    • One javascript function "play(type,scr)" is created. It plays media files (scr) with jPlayer.
  6. Import media files into Greenstone.  We need to preserve the file format of the media file to call the play(type,scr) function.  Therefore, we need to either extract file format automatically or enter the metadata manually.  "MP3 plugin" can handle mp3 file and create "ex.fileformat" metadata automatically.
     To extract other format automatically, a simple way is to config a new "UnknownPlugin", tick the "file_format" and enter the desired format for play function. Tick the "process_extension" and enter the real file extension (e.g. ogv) . Alternatively, tick "process_exp" and enter the regular express for the filename.  (e.g. m4v)  
    Process_extension File Format
    mp4(video)m4v
    mp4(audio)m4a
    m4vm4v
    m4am4a
    ogg(video)ogv
    ogg(audio)oga
    ogaoga
    ogvogv
    flvflv
    webmwebmv

    One can add some new process_extension if unconventional extensions are used. Due to the above limitation, for "*.mp4" or "*.ogg", one can use either all as video or audio. A mixture does not work.

Monday, August 1, 2011

A very bizarre behavior (bug) of "mv" command

I spend all the afternoon to find this bizarre behavior (bug) of "mv" command.

Let us say we have a directory of "source" with files "a" and "b", and we want to move it to "dest/sub/". So the final should look like "dest/sub/source/...". The command should be simple enough "mv source dest/sub".

Well, this works fine when "dest/sub" exists. When we only have "dest", a sub directory "sub" is automatically created, however, files ("a" and "b") are directly copied over instead of the directory "source".

It is like this in Linux (CentOS5.4) "GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)", as well as Mac OS 10.6 "GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin10.0)".

Monday, December 13, 2010

OpenID + Spring MVC 3 + Spring Security 3 + OpenID Selector

We decided to move our authentication into openID. Meanwhile, we upgrade our security from Spring Security 2 to 3. Adding OpenID to spring security is straightforward, add the following tag into "<http>" tag of security.xml
<openid-login login-page="/login.jsp" login-processing-url="/openlogin" user-service-ref="userDao"
                 authentication-failure-handler-ref="openIdAuthFailureHandler"/>

And OpenID will take over the authentication.

However, there are more to change for openID.

First and most obviously, we need a new login page takes user to openid provider instead of our own user/password checker. We chose openid-selector (1.2 currently) as the spring security choose it for demo. It is a fairly nice package. It allows one to choose from several openid providers (Google, Yahoo, AOL, OpenID, blogger, flicker, ...) Several things need to be done to use it:


  • However, it uses jQuery and has lots of conflict with prototype we are using. Fortunately, the conflict originates mostly from "$". I modify the two js file, replace all "$" with "jQuery" and add jQuery.noConflict(); in the beginning and firebug stops to complain.


  • There are several options in openid-jquery.js to play with. I use default mostly, except "no_sprite" to true as I have uncomment flickr and its not in the big sprite picture.


  • In the login page, add following into "<head"> tag (notice that openid-jquery.js should be add before openid-jquery-en.js)
    <script type="text/javascript" src="<c:url value="/scripts/jquery/openid-selector/js/openid-jquery.js" />"></script>
         <script type="text/javascript" src="<c:url value="/scripts/jquery/openid-selector/js/openid-jquery-en.js" />"></script>

    and add following into "<body">
    <script type="text/javascript">
            jQuery.noConflict();
            jQuery(document).ready(function(){
                openid.img_path="<c:url value='/scripts/jquery/openid-selector/images/'/>";
                openid.init("openid_identifier");
                jQuery("#openid_identifier").focus();
            });
        </script>

    Notice the openid.img_path has to be set outside if the openid-selector files are not put under the root.

These pretty much take care of login. But now we also need to modify signup process. Originally, you click signup and then put a new username/password. Now with openid, you first login with an openid from one of the providers, system found that you are not a user, and provide you a signup sheet. That is why in above <openid-login"> tag, we need a special "openIdAuthFailureHandler" for "authentication-failure-handler".

We can extend a spring handler for this purpose.
/*
 * To change this template, choose Tools | Templates
 * and open the template in the editor.
 */
package org.imirsel.nema.webapp.security;

import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.security.openid.OpenIDAuthenticationStatus;
import org.springframework.security.openid.OpenIDAuthenticationToken;
import org.springframework.security.web.DefaultRedirectStrategy;
import org.springframework.security.web.authentication.AuthenticationFailureHandler;
import org.springframework.security.web.authentication.SimpleUrlAuthenticationFailureHandler;

/**
 *  Customized {@link AuthenticationFailureHandler} that redirect to sign-up page
 * if the OpenID authentication succeeds, but the user name is not yet in local DB of the container
 * @author gzhu1
 */
public class OpenIDAuthenticationFailureHandler extends SimpleUrlAuthenticationFailureHandler {

    static private Log logger = LogFactory.getLog(OpenIDAuthenticationFailureHandler.class);

    @Override
    public void onAuthenticationFailure(HttpServletRequest request,
            HttpServletResponse response, AuthenticationException exception)
            throws IOException, ServletException {
        logger.error(exception, exception);
        if (exception instanceof UsernameNotFoundException
                && exception.getAuthentication() instanceof OpenIDAuthenticationToken
                && ((OpenIDAuthenticationToken) exception.getAuthentication()).getStatus().equals(OpenIDAuthenticationStatus.SUCCESS)) {
            DefaultRedirectStrategy redirectStrategy = new DefaultRedirectStrategy();
            request.getSession(true).setAttribute("USER_OPENID_CREDENTIAL", exception.getAuthentication().getPrincipal());
            // redirect to create account page

            logger.info("user (" + exception.getAuthentication().getPrincipal() + "," + exception.getExtraInformation() + ") is not found and redirect to signup.");
            redirectStrategy.sendRedirect(request, response, "/signup.html");

        } else {
            super.onAuthenticationFailure(request, response, exception);
        }
    }
}

and declare the bean in applicationContext.xml.
<bean id="openIdAuthFailureHandler" class="org.imirsel.nema.webapp.security.OpenIDAuthenticationFailureHandler">
        <property name="defaultFailureUrl" value="/login.jsp"/>
    </bean>

Of course, now I do not really have a password, so I generate a random string for password, because a password is needed somewhere else.

That is it. Now your user can sign-in with their Google, Yahoo, AOL, open-id account directly.

Tuesday, December 7, 2010

My Review of HP G62-340US 15.6" Laptop

Originally submitted at Staples

Now you can do more and have more fun without spending more. The HP G62 notebook PC features the latest technology and enhanced security right out of the box for the perfect balance of performance, connectivity, and worry-free computing. With its cl ean design and textured HP Imprint finish in char...


Solid buy

By zggame from urbana, IL on 12/7/2010

 

5out of 5

Pros: Quiet, Quality Display

Best Uses: Web Browsing, Video, Word Processing

Describe Yourself: Tech Savvy

Primary use: Personal

I bought at the sale for $379-$50 coupon for my parents. It has HDMI and webcam. They mostly just use skype, web browsing and watch online streaming TV/movie. It is perfect for their usage. The thrown in office starter is not bad. HDMI is easy for them to connect TV to watch video, better than VGA+audio port. Webcam works fairly well during skype. Overall, it is plenty for them. Great buy. Not too many bloatware. Norton is annoying for the trial. I uninstalled it and put a free Avast Personal. Office starter should be more than enough for some basic use.

(legalese)