Saturday, November 16, 2013

Using debootstrap and chroot to install Debian on a NAS (DS210+)

If you have a Linux based NAS with a powerful enough CPU, you may get an inkling to use it for more than what it was designed for, such as a general purpose server.  Well ... you can!!! ... and this blog tells you how to do it right!

I have a Synology DS210+, so this blog will be tailored around it, but the same approach should work for any NAS.

If you are thinking to use ipkg as outlined in the Synology Mod web-page,  DON'T DO IT!

You will quickly run into dependency conflict hell, especially as new DSM versions come out.  A much more elegant way is to use debootstrap and chroot to set up a parallel  Debian environment.  This has the benefit to compartmentalize your changes so that they do not interfere with the normal functionality of the original NAS OS + it opens the door to use any Debian package rather than the restricted few available under ipkg.

The DS210+ uses a Freescale CPU with an e500 PowerPC core.  In the Debian world, this CPU is not officially supported, but fortunately it has been accepted as an unofficial port under the powerpcspe architecture.  A big thank you to all the folks that made this happen ... the official page is here:

As mentioned on that page, the powerpcspe port makes use of packages in both unstable and unreleased depositories.  Debootsrap only supports fetching packages from one depository, so a special depository with the packages combined is provided.  This got me, and I spent significant time chasing phantom dependency problems due to not all packages being present in unstable ... including in some cases not having perl installed for properly running debootstrap on the target.

If someone has a pointer to a complete and detailed blog/tutorial on how to use multistrap (which is similar to debootstrap but allows multiple depositories) to generate and then unpack an installation archive, please add the link to the comments.  I searched and tried to put together the disjointed pieces for the powerpcspe architecture, but could not get it to work.

Enough talk, let's get going on this.

  1. First you need to enable SSH access on your NAS.  I suggest you also back up any critical data (even though chances are low that data will be impacted).  You assume total responsibility for your actions when you follow this guide.
  2. On a home PC, install VirtualBox with a current version of Debian.  There are plenty of tutorials out there, so I will not go into this.  A very easy one to follow is:
  3. In the Debian VirtualBox, open a Root Terminal and generate the Debian installation archive that will be installed on the target NAS:
    1. Install debootstrap if it is not already installed:

      apt-get install binutils debootstrap

    2. Go to your user directory and make the directory where you will generate the Debian environment (debian for this tutorial)

      cd ~
      mkdir debian
      cd debian

    3. Run debootstrap as below to get all the packages from a Debian depository.  Note that we are using the --foreign option, which tells debootstrap to just get and extract the packages, but it does not install them (this will be done on the target NAS device).  We are also using the --no-check-gpg option which tells deboostrap not to bother checking the packages against a Keyring.  The --arch option tells debootstrap that the target uses a CPU with a powerpcspe architecture.  Note that the depository used is the special /antcom.de one which combines packages from unstable and unreleased!

      debootstrap --foreign --no-check-gpg --arch powerpcspe unstable . http://antcom.de/powerpcspe/

    4. It should just run. If you run into any problems you will probably end up in dependency hell, so it is best to find an alternate distribution (check the debian page mentioned above to see if there are any updates). I ran it on the evening of Nov 12, 2013 and had no problems.
    5. Archive it all up now for transfer to the target:

      tar czf ../debian-stage2.tgz .

  4. Now, let's get the NAS ready to accept and finalize the Debian installation.  Log into your NAS through the SSH shell and do the following:
    1. Select the location where you would like to install the Debain installation.  In this case I will be installing it in /volume1/@debian/wheezy/ (wheezy is the name of the current Debian distribution):

      cd /volume1
      sudo mkdir /@debian
      cd @debian
      sudo mkdir wheezy

    2. Copy the debian-stage2.tgz file into the above wheezy directory.
    3. Extract the archive, chroot to the extracted archive and run the second stage of debootstrap, which installs all the packages on the target.  Then exit the shell for some other housekeeping:

      cd /volume1/@debian/wheezy/
      sudo tar xzf debian-stage2.tgz
      sudo chroot /volume1/@debian/wheezy /bin/bash
      sudo /debootstrap/debootstrap --second-stage
      exit

  5. Ok, now you have Debian installed on your box in parallel with the original OS; albeit this distribution is a bit dated.  No worries, we will bring it up to date next.  For this, you have to add current depositories to the sources.list file:
    1. Update /volume1/@debian/wheezy/etc/apt/sources.list with your favorite text editor with pointers to the debian-ports depositories (ie sudo vi /volume1/@debian/wheezy/etc/apt/sources.list). Add the following lines to the file (it should be blank to begin with):

      deb http://ftp.de.debian.org/debian-ports unstable main
      deb http://ftp.de.debian.org/debian-ports unreleased main
      deb http://ftp.de.debian.org/debian-ports experimental main

      (Note: I did not put in any deb-source entries, maybe someone can comment if they would be worthwhile to add)
    2. After saving the file, go to your home directory:

      cd ~

    3. Make two script files that will come in handy to run Debian. The first file is called initWheezy.sh and will be used to set up the environment every time after a power up.  It should have the following content:

      mount -o bind /volume1/@debian /opt/debian
      mount -o bind /dev /opt/debian/wheezy/dev
      mount -o bind /proc /opt/debian/wheezy/proc
      mount -o bind /sys /opt/debian/wheezy/sys
      cp /etc/resolv.conf /opt/debian/wheezy/etc/
      cp /etc/hosts /opt/debian/wheezy/etc/
      chroot /opt/debian/wheezy /bin/bash

    4. The second file is called wheezy.sh, it will be used to just get into the debian installation after exiting it (but not rebooting the NAS):

      chroot /opt/debian/wheezy /bin/bash

    5. Now make the scripts executable and run the first script to set up the chroot environment:

      sudo chmod 777 initWheezy.sh
      sudo chmod 777 wheezy.sh
      sudo ./initWheezy.sh

    6. We are now running within the Debian environment.  Before proceeding with upgrading the packages, we must upgrade the debian keyring so that apt may verify the packages being installed.  First you need to try and get the updated package list (first line below); this should fail and give you an error with a missing public key.  You take the key provided and plug it into the following commands replacing the put_key_here placeholder with the key number:

      apt-get update
      # should get signature here with missing PUBKEY
      apt-get install debian-keyring
      gpg --keyserver pgp.mit.edu --recv-keys put_key_here
      gpg --armor --export put_key_here | apt-key add -

    7. Now we are ready to perform the Debian update. Execute the following commands:

      apt-get update
      apt-cache gencaches
      apt-get upgrade
      apt-get dist-upgrade

      As before, everything should just run. You may want to run the update, upgrade, and dist-upgrade triplet a second time to make sure all packages that could be upgraded where. You may have some packages that were held back due to dependency reasons and not upgraded, this should not be an issue.
  6. Now you should have a fully functional Debian distribution running in parallel with your NAS OS. Before going all out, you may want to do some minor clean-up and archive your installation in case you screw up during a problematic package download.  If this happens, then you can just delete the wheezy directory and re-install it in a couple of minutes. I suggest you do this every time you get to a point where everything is stable and you have installed the  desired packages.
    1. Set up a user, so that once you run chroot you can "login user_name" before running headless X11 VNC servers or anything else, so that you are not running them from root (but still give it sudo powers):

      adduser user_name
      adduser user_name sudo

    2. Now do a quick clean-up, including cleaning up the downloaded archives:

      rm debian-stage2.tgz
      apt-get clean
      exit

    3. Now you need to power cycle the box. After that, archive the current installation by doing the following:

      cd /volume1/@debian/wheezy
      sudo tar czf ../wheezy_date.tgz .

  7. Now you are ready to use it as you please. Re-run the script to initialize the chroot environment (you need to run it after every power cycle):

    sudo ./initWheezy.sh

    use the other script (sudo ./wheezy) after you exit or logout and need to get back into it.

At this point you can install any available Debian package, only limited by what is in the distribution and the resources of your NAS. The DS210+ has a 1GHz e500 single core CPU with 512MB of RAM, which is not much by todays standards; but I was pretty amazed at how smoothly everything ran. I added a headless VNC servers so that I could give it a desktop feel. Using the above distribution, GNOME was not available, so I ended up using vnc4server with Xfce4 desktop instead. I also installed Jetty and Tomcat Java servers (you must use a different port than is being used by your NAS OS, though).

Use Aptitude instead of apt or dpkg, as it works really well in identifying and resolving any dependency conflicts.

This really opens up the doors of what you can use these NAS boxes for ... just awesome!

Below are a couple of other blogs that provided the inspiration and helped me to put this blog together:

Also, as far as finding stable Debian distributions to upgrade to, look at the following locations as potential alternatives (the one used in the blog is a mirror of http://ftp.debian-ports.org/debian/):

--- Good Luck! ---

GWT CompositeCell improvement to allow complex layouts

... Improved CompositeCell class based on HTML id attributes allows the Cells to use complex layouts ...

GWT Cells are a pretty cool concepts that allows for very quick rendering of widgets by generating a SafeHTML string of all the underlying HTML code, and then rendering it to the innerHTML of a container element.  By just building a large string, execution during rendering remains purely in Javascript, thus avoids time consuming repeated interactions with the DOM.

The downside is that the Cell widget does not have a direct references to the inner DOM elements, hence manipulation of the elements after rendering is more problematic and time consuming.  With UiRenderer added in 2.5, GWT has put in place simple hooks for allowing these interactions. Now you can easily handle events, get individual DOM elements, re-render individual parts or the whole cell, and get obfuscated class name handles.  so you can now easily implement fairly sophisticated Cells. Albeit, the implementations require significant boiler plate code, which hopefully will be done away with in a future GWT release.

After spending significant cycles implementing Cells with all the bells and whistles, there will quickly come the need to combine some of these into a super Cell.  If the underlying Cell widgets are not simple, generating such a supercell by using a custom cell and combining all the logic can be a daunting task. GWT does have a CompositeCell class to do this, but as far as I can tell, it has a couple of large deficiencies:
  • The Cells must be placed sequentially within a container tag, hence limiting the layout options to rows or columns.  Deviating from this breaks the event capability of the cell.
  • CompositeCell extends AbstractCell, which does not have the View Data object available in AbstractEditableCell.

The CompositeCellWithId class below addresses these shortcomings, and provides full layout flexibility while retaining event functionality.  The main difference is in the way that the Children Cells are found after rendering.  In the original GWT CompositeCell class, these are found by assuming a relative structure, and navigating through that.  In the CompositeCellWithId class, HTML id attributes are added to the Child container elements; this allows the container elements to be quickly found by using the Element#getElementById() method.  This is similar to how GWT already provides DOM element references for elements identified through UiRenderer with the ui:field attribute.  I wish that the resulting renderer provided the base id used for these so it can be re-used during rendering, but it does not.

In addition, the improved class adds the following features/improvements:
  • Event functionality is fully retained independent of Composite Cell layout.
  • A View Data object can be added to the composite.  When not using it, just pass in a dummy class in the generic (ie Integer).
  • Allows the HTML tags for composite cell element and the Children wrapper elements to be redefined.  This way one can generate an efficient table without wasted meaningless elements.
  • Allows the composite cell to be built up directly, rather then using a predefined list, to improve code flow.
  • It finds the target Child cell directly by parsing up from the target, rather than querying each cell to see if they contain the target.
  • It allows the Cell.Context to be changed as it is propagated by overriding createContext().
  • It generates the id attribute by using GWT’s Document.get().createUniqueId(); one can be override this to provide deterministic handles.
  • It provides a SimpleHasCell class to help streamline the Cell additions to the Composite.
  • Id attribute manipulations are done in a separate inner class, so that if a different behavior is desired, the class can be extended and re-used across multiple Composite Cells.
  • The base HTML id attribute can be read out while overriding one of the render methods, so that other code can refer to it.  Note that in the default implementation the id is changed every time the cell is rendered.
  • A valueUpdater can be added to the CompositeCell container that will also be called after one of the underlying child cells is updated (same as in the original CompositeCell)

One general comment, if you are looking to just use composite cells as non-interactive HTML stamps, you are better off using a custom cell as it removes an additional layer of hierarchy that adds overhead, especially for events.

Be cognizant that editable Cells (such as EditTextCell) maintain their edit state in a Map until the changes are committed.  As a result, one has to make sure that the same Cell instance (ie new EditTextCell) is not used in multiple places that would get the same key, as this would result in the value at one location contaminating that of another.  As shown in the example below, each Composite Cell editable child gets their own Cell instance.  One can alter this behavior by overriding the createContext() method and changing what is used by the key parameter.  See the GWT Cell class documentation for more details.

For reference, the following is how some of the Cell Widgets use the context:

IndexColumnKeySub-Index
CellTableHeader/Footer: row
Data: Row + Start
ColumngetKey() from KeyProvider, or
the row object if no KeyProvider
subrow
DataGridSame as CellTableColumnSame as CellTablesubrow
CellListRow + Start0Same as CellTable0
CellWidget00Same as CellTable0

Please feel free to leave comments if this was helpful or if there are other approaches that would be better.  I really like the cell concept, but I think it would benefit from a systematic centralized controller approach where all things cell can be coordinated and recorded to allow for easier document wide implementation and standard Javascript interaction.  Such a configuration, would really allow only one cell of each type to be instantiated per document.

How to use this code:
The code snippet below generates the following raw cell format, showing that it can use fully interactive cells, such as EditTextCell, TextInputCell, and DatePickerCell, in a table layout within a Composite Cell:

Below is the code snippet of how this class generate the nicely formatted Composite Cell.  Notice that we are not using the View data State, so one can pass in any Type into the generic (I used Integer below).
public ViewDashboard() {

    /* CompositeCellWithId example */
    // Create the example database list
    PersonComposite person1 = new PersonComposite("John","Smith","Software Developer", new Date(), new Date());
    PersonComposite person2 = new PersonComposite("Jane","Doe","Hardware Developer", new Date(), new Date());
    List<PersonComposite> personList = Arrays.asList(person1,person2);
  
    // Create the composite widget
    CompositeCellWithId<PersonComposite,Integer> cell2 = new CompositeCellWithId<PersonComposite,Integer>(){
            @Override
            public void renderListOfCells(Context containerContext, PersonComposite containerValue, SafeHtmlBuilder sb, String containerBaseId) {
                sb.appendHtmlConstant("<fieldset><legend>This is a Composite Cell With Id</legend>");
                sb.appendHtmlConstant("<table><tr><td>First Name:</td>");
                renderIndividualCell(containerContext, containerValue, sb, getHasCell(0), 0, containerBaseId);
                sb.appendHtmlConstant("</tr><tr><td>Last Name:</td>");
                renderIndividualCell(containerContext, containerValue, sb, getHasCell(1), 1, containerBaseId);
                sb.appendHtmlConstant("</tr><tr><td>Title Date:</td>");
                renderIndividualCell(containerContext, containerValue, sb, getHasCell(2), 2, containerBaseId);
                sb.appendHtmlConstant("</tr><tr><td>Hire Date:</td>");
                renderIndividualCell(containerContext, containerValue, sb, getHasCell(3), 3, containerBaseId);
                sb.appendHtmlConstant("</tr><tr><td>Fire Date:</td>");
                renderIndividualCell(containerContext, containerValue, sb, getHasCell(4), 4, containerBaseId);
                sb.appendHtmlConstant("</tr></table>");
                sb.appendHtmlConstant("</fieldset>");
            }
        };
    // Change Container and Cell tag from default.
    cell2.setCompositeContainerTag("span").setCellContainerTag("td");
 
    // Add the individual cells to the composite
    cell2.add(new SimpleHasCell<PersonComposite, String>(new EditTextCell()) {
            @Override public String getValue(PersonComposite object) { return object.firstName; }
            @Override public void update(int index, PersonComposite object, String value) { object.firstName = value; }
        });
    cell2.add(new SimpleHasCell<PersonComposite, String>(new EditTextCell()) {
            @Override public String getValue(PersonComposite object) { return object.lastName; }
            @Override public void update(int index, PersonComposite object, String value) { object.lastName = value; }
        });
    cell2.add(new SimpleHasCell<PersonComposite, String>(new TextInputCell()) {
            @Override public String getValue(PersonComposite object) { return object.title; }
            @Override public void update(int index, PersonComposite object, String value) { object.title = value; }
        });
    cell2.add(new SimpleHasCell<PersonComposite, Date>(new DatePickerCell()) {
            @Override public Date getValue(PersonComposite object) { return object.hireDate; }
            @Override public void update(int index, PersonComposite object, Date value) { object.hireDate = value; }
        });
    cell2.add(new SimpleHasCell<PersonComposite, Date>(new DatePickerCell()) {
            @Override public Date getValue(PersonComposite object) { return object.fireDate; }
            @Override public void update(int index, PersonComposite object, Date value) { object.fireDate = value; }
        });
    cell2.initCell();
  
    // Generate the CellList
    cellList = new CellList<PersonComposite>(cell2);
    cellList.setRowCount(personList.size(), true);
    cellList.setRowData(personList);
  
    initWidget(uiBinder.createAndBindUi(this));
}
Here is the CompositeCellWithId class:
It looks a lot bigger then it is, as there are a lot of comments.
/**
 * Copyright 2010 Google Inc.
 * 
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License. You may obtain a copy of
 * the License at
 * 
 * http://www.apache.org/licenses/LICENSE-2.0
 * 
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */
package com.sneclacson.addressbook.client.AppAbstracts;

import com.google.gwt.cell.client.AbstractEditableCell;
import com.google.gwt.cell.client.Cell;
import com.google.gwt.cell.client.FieldUpdater;
import com.google.gwt.cell.client.HasCell;
import com.google.gwt.cell.client.ValueUpdater;
import com.google.gwt.dom.client.Document;
import com.google.gwt.dom.client.Element;
import com.google.gwt.dom.client.EventTarget;
import com.google.gwt.dom.client.NativeEvent;
import com.google.gwt.regexp.shared.MatchResult;
import com.google.gwt.regexp.shared.RegExp;
import com.google.gwt.safehtml.shared.SafeHtmlBuilder;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

/**
 * <p>
 * A {@link Cell} that is composed of other {@link Cell}s.
 * Uses com.google.gwt.cell.client.CompositeCell from GWT 2.5.1 as a starting point.
 * </p><p>
 * The advantages of this composite cell implementation versus the original CompositeCell are:
 * <ul>
 * <li> This composite cell can be rendered in an arbitrary shape, ie it is 
 * not restricted to a simple row or column structure.  This is possible as each 
 * child cell is wrapped in a cell container with a unique cell ID
 * These containers are easily retrieve by getElementById.</li>
 * <li> The child cell functionality is fully retained (ie events), even when 
 * the cell is laid out in an arbitrary shape. </li>
 * <li> The composite cell can have ViewData (from AbstractEditableCell).</li>
 * <li>The container cell and the underlying child cells can be assigned arbitrary
 * HTML tag elements.  This comes in handy if one want to generate a table, the
 * container can be a TR element, while the children can be TD elements.</li>
 * <li>Allows the <code>List<HasCell>></code> to be built within the class,
 * after the individual cells are added, call the {@link #initCell(String...)} method.</li>
 * <li>The target cell is found by traversing the DOM from the target element, rather
 * than asking all the Cells if they contain the target (more scalable).</li>
 * </ul>
 * </p><p>
 * Once initialized, one should not change the value of its members.  Cells cannot
 * be added or removed dynamically.
 * </p><p>
 * All HasCells "items" must be used exactly once in the composite cell.  If the same
 * HasCells index location is used more than once, there will be duplicate HTML id, and if
 * an index is not used, getElementById will not find the element.
 * </p><p>
 * A new container tag HTML id is generated every time the cell is rendered.
 * To provide a deterministic Container HTML Id rather than the auto generated one,
 * override the {@link IdProcessor#getNewUniqueId(com.google.gwt.cell.client.Cell.Context, Object, SafeHtmlBuilder)} method.
 * To get access to the rendered id, one can override the {@link #renderListOfCells(com.google.gwt.cell.client.Cell.Context, Object, SafeHtmlBuilder, String)} method.
 * </p><p>
 * {@link #createCellContext(com.google.gwt.cell.client.Cell.Context, Object, HasCell, int, String)} can
 * be overriden to manipulate the context as it is passed down.  This could be useful if a deterministic
 * HTML id attribute is used, then the id can be passed in instead of the Object key, thus allowing the 
 * same cell to be reused within the same CompositeCell and across widgets.
 * </p><p>
 * WishList:
 * <ul>
 * <li> UiRenderer as of 2.5.1 does not support adding children cells.  This would
 * be really nice if it can be used in combination with this. </li>
 * </ul>
 * </p><p>
 * Fund a bug in EditTextCell while testing this: Edit the text, deselect, select,
 * deselect, select ... and the Edit is lost.  It can be seen in Showcase Cell Sampler.
 * </p>
 * 
 * @param <C> the type that this Cell represent.  The data from this object is extracted by
 *            the HasData#getValue method.
 * @param <V> the data type of the view data state of the container Cell, not the underlying cells it holds
 */
public class CompositeCellWithId<C,V> extends AbstractEditableCell<C,V> {
    protected final static String CELL_TAG = "_celltag";
    protected final static String PARAMETER_SEPARATOR = "_P";
    protected final static String PARAMETER_END = "_";
 
    /** This class provides convenience methods for the HTML tag id manipulation.
     * One can override the various methods to change the cell Id format, of special significance
     * is the {@link #getNewUniqueId(com.google.gwt.cell.client.Cell.Context, Object, SafeHtmlBuilder)}
     * method that can be overriden to change how the cell BaseId is generated (ie maybe from
     * the context key object) or some other deterministic mean.  */
    public static class IdProcessor {

        protected static RegExp regexExtractBaseId = RegExp.compile("(.*?)" + PARAMETER_SEPARATOR);
        protected static RegExp regexExtractAll = RegExp.compile( "(?:" + PARAMETER_SEPARATOR + "(.*?)" + PARAMETER_END + ")+?" );
        protected static RegExp regexExtractFirst = RegExp.compile( PARAMETER_SEPARATOR + "(.*?)" + PARAMETER_END );
  
        // ========== Constructors ==========
        public IdProcessor(){}
  
        /** Returns a Unique ID String.  Override this method if a deterministic approach is desired */
        public <C> String getNewUniqueId(Cell.Context context, C containerValue, SafeHtmlBuilder sb) {
            return Document.get().createUniqueId();
        }
 
        /** Returns the Primary Id.  If there is no Primary ID to be extracted, it returns self. */
        public String getBaseId(String cellId){
            MatchResult matcher = regexExtractBaseId.exec(cellId);
            return (matcher.getGroupCount() != 2) ? cellId : matcher.getGroup(1);
        }  
  
        /** Returns all the parameters directly in the MatchResult. */
        public MatchResult getAllParamenters(String cellId){
            MatchResult matcher = regexExtractAll.exec(cellId);
            return matcher;
        }
  
        /** Returns empty string if the Index is not found, otherwise it returns the first extracted parameter. */
        public String getFirstParamenter(String cellId){
            MatchResult matcher = regexExtractFirst.exec(cellId);
            return (matcher.getGroupCount() != 2) ? "" : matcher.getGroup(1);
        }
  
        public String appendParameter(int param, String cellId){
            return (cellId + PARAMETER_SEPARATOR + param + PARAMETER_END);
        }
  
        public String appendParameter(String param, String cellId){
            return (cellId + PARAMETER_SEPARATOR + param + PARAMETER_END);
        }

        public boolean isSameBaseId(String baseId, String testString){
            return testString.startsWith(baseId);
        }
    }
 
    /** Convenience HasCell class implementation that leaves only getValue and update methods to be overriden */
    static public abstract class SimpleHasCell<C,Z> implements HasCell<C,Z>, FieldUpdater<C,Z>{

        Cell<Z> cell;
  
        public SimpleHasCell(Cell<Z> cell) {
            this.cell = cell;
        }
  
        @Override
        public Cell<Z> getCell() { return cell; }

        @Override
        public FieldUpdater<C,Z> getFieldUpdater() {
            return this;
        }
    }
 
    /** The events consumed by this cell. */
    protected Set<String> consumedEvents;

    /** Indicates whether or not this cell depends on selection. */
    protected boolean dependsOnSelection;

    /** Indicates whether or not this cell handles selection. */
    protected boolean handlesSelection;

    /**
     * The cells that compose this {@link Cell}.
     * 
     * NOTE: Do not add add/insert/remove hasCells methods to the API for this list. This cell
     * assumes that the index of the cellParent corresponds to the index in the
     * hasCells array.
     */
    protected List<HasCell<C,?>> hasCells;
 
    /** Used to build the hasCells list before finalizing it. */
    protected List<HasCell<C,?>> hasCellsBuilder;
  
    /** Tag for the composite container, the default is div */
    protected String compositeContainerTag = "div";
  
    /** Tag for the individual cell containers, the default is span */
    protected String cellContainerTag = "span";
  
    /** Instantiated so that it can use sub-classes. */
    protected static IdProcessor idProcessor;
  
    // ========= Constructors ==========

    /** Generates the basic Composite Cell without an attached HasCell List.
     * One needs to add the HasCell elements and then call {@link #initCell(String...)}.
     * Note: as upper widgets cache information, this is only done to use this
     * as a builder and cannot be used to redefine it on the fly.
     */
    public CompositeCellWithId() {
        hasCellsBuilder = new ArrayList<HasCell<C,?>>();
    }
 
    /** A new {@link CompositeCellWithId}.
     * @param hasCells the cells that makeup the composite
     * @param sinkEvents the events that this composite cell can sink (do NOT include the child cells events). */
    public CompositeCellWithId(List<HasCell<C,?>> hasCells, Set<String> sinkEvents) {
        initCell(hasCells, sinkEvents);
    }
 
    /** A new {@link CompositeCellWithId}.
     * @param hasCells the cells that makeup the composite
     * @param sinkEvents the events that this composite cell can sink (do NOT include the child cells events). */
    public CompositeCellWithId(List<HasCell<C,?>> hasCells, String... sinkEvents) {
        initCell(hasCells, (sinkEvents.length == 0) ? null : new HashSet<String>(Arrays.asList(sinkEvents)));
    }
    
    /** Initializes the Cell fields using a Set of Strings. Used only when {@link #CompositeCellWithId()} constructor is used. */
    public CompositeCellWithId<C,V> initCell(Set<String> sinkEvents) {
        initCell(hasCellsBuilder, sinkEvents);
        return this;
    }

    /** Initializes the Cell fields using an array of Strings. Used only when {@link #CompositeCellWithId()} constructor is used. */
    public CompositeCellWithId<C,V> initCell(String... sinkEvents) {
        initCell(hasCellsBuilder, (sinkEvents.length == 0) ? null : new HashSet<String>(Arrays.asList(sinkEvents)));
        return this;
    }
 
    protected CompositeCellWithId<C,V> initCell(List<HasCell<C,?>> hasCells, Set<String> sinkEvents) {
        // Selects how the HTML ID of the composite cell will be manipulated.
        if (idProcessor == null) idProcessor = new IdProcessor();
      
        // Create a new array so cells cannot be added or removed after init.
        this.hasCells = new ArrayList<HasCell<C,?>>(hasCells);

        // Get the consumed events and depends on selection.
        Set<String> theConsumedEvents = null;
        for (HasCell<C,?> hasCell : hasCells) {
            Cell<?> cell = hasCell.getCell();
            Set<String> events = cell.getConsumedEvents();
            if (events != null) {
                if (theConsumedEvents == null) {
                    theConsumedEvents = new HashSet<String>();
                }
                theConsumedEvents.addAll(events);
            }
            if (cell.dependsOnSelection()) {
                dependsOnSelection = true;
            }
            if (cell.handlesSelection()) {
                handlesSelection = true;
            }
        }
        // Add events for this composite cell
        if (sinkEvents != null) {
            if (theConsumedEvents == null) theConsumedEvents = new HashSet<String>();
            theConsumedEvents.addAll(sinkEvents);
        }
        if (theConsumedEvents != null) {
            this.consumedEvents = Collections.unmodifiableSet(theConsumedEvents);
        }
        return this;
    }
    
    /** Add an individual hasCell during the Composite Cell initialization process.  Cannot be used after the {@link #initCell(String...)} is called.*/
    public CompositeCellWithId<C,V> add(HasCell<C,?> hasCell) {
        hasCellsBuilder.add(hasCell);
        return this;
    }
    
    /** Defines how the Cell.Context passed to the cell is generated.  Separated out so that it can be overriden. */
    protected <X> Context createCellContext(Context containerContext, C containerValue, HasCell<C,X> hasCell, int index, String containerBaseId){
        return containerContext;
    }
   
    @Override
    public boolean dependsOnSelection() {
        return dependsOnSelection;
    }

    @Override
    public Set<String> getConsumedEvents() {
        return consumedEvents;
    }

    @Override
    public boolean handlesSelection() {
        return handlesSelection;
    }

    @Override
    public boolean isEditing(Context containerContext, Element containerParent, C containerValue) {
        String containerBaseId = containerParent.getFirstChildElement().getId();
        for (int index = 0; index < hasCells.size(); index++){
            Element cellParent = getChildElementById(containerBaseId, index);
            Context cellContext = createCellContext(containerContext, containerValue, hasCells.get(index), index, containerBaseId);
            if(isEditingImpl(cellContext, cellParent, containerValue, hasCells.get(index))) {
                return true;
            }
        }
        return false;
    }

    /** Finds the child cell with the event target and propagates the event. */
    @Override
    public void onBrowserEvent(Context containerContext, Element containerParent, C containerValue, NativeEvent event, ValueUpdater<C> containerValueUpdater) {
        EventTarget eventTarget = event.getEventTarget();
        if (!Element.is(eventTarget)) return;
        Element target = eventTarget.cast();
        String containerBaseId = containerParent.getFirstChildElement().getId();
        Element targetChildWrapper = findTargetChildWrapper(target, containerBaseId);
        if (targetChildWrapper == null) return;
        int index =  Integer.parseInt(idProcessor.getFirstParamenter(targetChildWrapper.getId()));
        Context cellContext = createCellContext(containerContext, containerValue, hasCells.get(index), index, containerBaseId);
        onBrowserEventImpl(cellContext, targetChildWrapper, containerValue, event, containerValueUpdater, hasCells.get(index));
    }
    
    /** get Child Element by ID. */
    public Element getChildElementById(String cellId, int index) {
        Element result = Document.get().getElementById(idProcessor.appendParameter(index, cellId));
        assert (result != null) : "Could not find HTML tag id: " + idProcessor.appendParameter(index, cellId);
        return result;
    }
    
    /** Finds the child that contains the target element */
    protected Element findTargetChildWrapper(Element target, String baseId) {
        Element nextElement = target;
        while (nextElement != null) {
            if (nextElement.hasAttribute(CELL_TAG + "child"))  {
                if ( idProcessor.isSameBaseId(baseId, nextElement.getId()) ) {
                    return nextElement;
                }
            }
            nextElement = nextElement.getParentElement();
        }
        return null;
    }
    
    /**
     * Render the full composite as HTML into a {@link SafeHtmlBuilder}, suitable
     * for passing to {@link Element#setInnerHTML} on a container element.
     * <p>
     * Note: If your cell contains natively focusable elements, such as buttons or
     * input elements, be sure to set the tabIndex to -1 so that they do not steal
     * focus away from the containing widget.
     * </p><p>
     * The full composite must be wrapped using {@link #startCompositeCellTag(String, SafeHtmlBuilder, String)}
     * and {@link #endTag(SafeHtmlBuilder, String)}.
     * </p><p>
     * The individual cells need to be rendered using {@link #renderIndividualCell(com.google.gwt.cell.client.Cell.Context, Object, SafeHtmlBuilder, HasCell, int, String)}
     * to ensure proper wrapping so that the composite can propagate events.
     * </p>
     * @param containerContext the {@link com.google.gwt.cell.client.Cell.Context Context} of the cell
     * @param containerValue the cell value to be rendered
     * @param sb the {@link SafeHtmlBuilder} to be written to
     */

    @Override
    public void render(Context containerContext, C containerValue, SafeHtmlBuilder sb) {
        String containerBaseId = idProcessor.getNewUniqueId(containerContext, containerValue, sb);
        startCompositeCellTag(containerBaseId, sb, compositeContainerTag);
        renderListOfCells(containerContext, containerValue, sb, containerBaseId);
        endTag(sb, compositeContainerTag);
    }
    
    /** Iterates through the cells to add them one by one, separated from {@link #render(com.google.gwt.cell.client.Cell.Context, Object, SafeHtmlBuilder)}
     *  to make it easier to override and add headers/footers. */
    public void renderListOfCells(Context containerContext, C containerValue, SafeHtmlBuilder sb, String containerBaseId) {
        for (int index = 0; index < hasCells.size(); index++) {
            renderIndividualCell(containerContext, containerValue, sb, getHasCell(index), index, containerBaseId);
        }
    }

    @Override
    public boolean resetFocus(Context containerContext, Element containerParent, C containerValue) {
        String containerBaseId = containerParent.getFirstChildElement().getId();
        for(int index=0; index < hasCells.size(); index++) {
            // The first child that takes focus wins.  Only one child should ever be in edit mode, so this is safe.
            Context cellContext = createCellContext(containerContext, containerValue, hasCells.get(index), index, containerBaseId);
            Element cellParent = getChildElementById(containerBaseId, index);
            if ( resetFocusImpl(cellContext, cellParent, containerValue, hasCells.get(index)) ) return true;
        }
        return false;
    }

    @Override
    public void setValue(Context containerContext, Element containerParent, C containerValue) {
        String containerBaseId = containerParent.getFirstChildElement().getId();
        for (int index=0; index < hasCells.size(); index++) {
            Context cellContext = createCellContext(containerContext, containerValue, hasCells.get(index), index, containerBaseId);
            Element cellParent = getChildElementById(containerBaseId, index);
            setValueImpl(cellContext, cellParent, containerValue, hasCells.get(index));
        }
    }

    /**
     * Render one of the composite cell as HTML into a {@link SafeHtmlBuilder}, suitable
     * for passing to {@link Element#setInnerHTML} on a container element.
     * <p>
     * Note: If your cell contains natively focusable elements, such as buttons or
     * input elements, be sure to set the tabIndex to -1 so that they do not steal
     * focus away from the containing widget.
     * </p><p>
     * Each individual cell must be wrapped using {@link #startChildCellWrapperTag(String, SafeHtmlBuilder, String, int)} 
     * and {@link #endTag(SafeHtmlBuilder, String)}.
     * </p>
     * @param cellContext the {@link com.google.gwt.cell.client.Cell.Context Context} of the cell
     * @param containerValue the cell value to be rendered
     * @param sb the {@link SafeHtmlBuilder} to be written to
     * @param hasCell contains the cell to be rendered within this composite.
     * @Param index the index into the hasCells Array.
     */
    protected <X> void renderIndividualCell(Context containerContext, C containerValue, SafeHtmlBuilder sb, HasCell<C,X> hasCell, int index, String containerBaseId) {
        Cell<X> cell = hasCell.getCell();
        Context cellContext = createCellContext(containerContext, containerValue, hasCell, index, containerBaseId);
        // update index and key fields, use the column and subIndex field from the container context
        startChildCellWrapperTag(containerBaseId, sb, cellContainerTag, index);
        cell.render(cellContext, hasCell.getValue(containerValue), sb);
        endTag(sb, cellContainerTag);
    }
    
    // Same as GWT 2.5.1 CompositeCell
    protected <X> boolean isEditingImpl(Context cellContext, Element cellParent, C object, HasCell<C,X> hasCell) {
         return hasCell.getCell().isEditing(cellContext, cellParent, hasCell.getValue(object));
    }  

    // Same as GWT 2.5.1 CompositeCell
    protected <X> void onBrowserEventImpl(final Context cellContext, Element cellParent, final C containerValue, NativeEvent event, final ValueUpdater<C> containerValueUpdater, final HasCell<C,X> hasCell) {
        Cell<X> cell = hasCell.getCell();
        String eventType = event.getType();
        Set<String> cellConsumedEvents = cell.getConsumedEvents();
     
        // If this sub-cell doesn't consume this event.
        if (cellConsumedEvents == null || !cellConsumedEvents.contains(eventType)) return;
     
        // Call the underlying cell event; note that the HasCell FieldUpdater<C,X> is a superset of the Cell ValueUpdater<X>, so wrapper needs to be added.
        final FieldUpdater<C,X> fieldUpdater = hasCell.getFieldUpdater();
        ValueUpdater<X> tempValueUpdater = null;  
        if (fieldUpdater != null) {
        tempValueUpdater = new ValueUpdater<X>() {
                @Override
                public void update(X value) {
                    fieldUpdater.update(cellContext.getIndex(), containerValue, value);
                    // Passes in the modified object, presumably so that the logic to update the database is only required at the composite level.
                    if (containerValueUpdater != null) containerValueUpdater.update(containerValue);
                }
            };
        }
        cell.onBrowserEvent(cellContext, cellParent, hasCell.getValue(containerValue), event, tempValueUpdater);
    }

    // Same as GWT 2.5.1 CompositeCell
    protected <X> boolean resetFocusImpl(Context cellContext, Element cellParent, C containerValue, HasCell<C,X> hasCell) {
        X cellValue = hasCell.getValue(containerValue);
        return hasCell.getCell().resetFocus(cellContext, cellParent, cellValue);
    }

    // Same as GWT 2.5.1 CompositeCell
    protected <X> void setValueImpl(Context cellContext, Element cellParent, C containerValue, HasCell<C,X> hasCell) {
        hasCell.getCell().setValue(cellContext, cellParent, hasCell.getValue(containerValue));
    }  
  
    /** Append the composite container opening Tag with predefined unique id. Tag from {@link #setCellContainerTag(String)}. */
    protected SafeHtmlBuilder startCompositeCellTag(String cellId, SafeHtmlBuilder sb, String tag) {
        return sb.appendHtmlConstant("<" + tag + " id='" + cellId + "' " + CELL_TAG + "='true'>");
    }
 
    /** Add the composite container opening Tag with predefined unique id. Tag from {@link #setCellContainerTag(String)}. */
    protected SafeHtmlBuilder startChildCellWrapperTag(String cellId, SafeHtmlBuilder sb, String tag, int index) {
        return sb.appendHtmlConstant("<" + tag + " id='" + idProcessor.appendParameter(index, cellId) + "' " + CELL_TAG + "child='true'>");
    }

    /** Add the generic end Tag. */
    protected SafeHtmlBuilder endTag(SafeHtmlBuilder sb, String tag){
        return sb.appendHtmlConstant("</" + tag + ">");
    }
  
    /** Sets the Tag for the composite container, the default is div. */
    public CompositeCellWithId<C,V> setCompositeContainerTag(String c) { compositeContainerTag = c; return this; }
    public String getCompositeContainerTag() { return compositeContainerTag; }
    
    /** Sets the Tag for the cell container, the default is span. */
    public CompositeCellWithId<C,V> setCellContainerTag(String c) { cellContainerTag = c; return this; }
    public String getCellContainerTag() { return cellContainerTag; }
    public HasCell <C,?> getHasCell(int index) { return hasCells.get(index); }
}