Introduction

Over the last few weeks I have been working on getting dremio-oss up and running. It’s an interesting challenge for me because I’ve done very little java. So there is definitely a lot to learn here. After doing this build a few times under a much more constrained environment, I decided to run a build out on Google cloud to see how long it took and shake out any flaws in my build script. When I say build script, at this point, it’s just a bunch of commands from memory…

Since I haven’t been back into GCP in a while, and I needed a relatively sterile system to build on, I decided to load this in on a Compute Engine and refresh my learning there.

At a gcp system prompt

I have to setup several tools to start with. Dremio depends on java, maven and npm. Oh, I also included fish and Emacs in because I tend to fall back to them for text editing and command line support.

apt-get

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install openjdk-8-jdk maven screen fish emacs

java version check

user@go-devel ~> java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

maven version check

user@go-devel ~> mvn -version
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.9.0-8-amd64", arch: "amd64", family: "unix"

npm

It took me a little while to understand the npm dependency. I finally gave up on apt-get approach and opted to pull the package from what appears to be a known reputable source.

mkdir $HOME/packages
curl https://nodejs.org/dist/v10.15.3/node-v10.15.3-linux-x64.tar.xz > $HOME/packages/node-v10.15.3-linux-x64.tar.xz 
pushd $HOME/packages
tar -xf node-v10.15.3-linux-x64.tar.xz 
popd
set -x PATH $PATH $HOME/packages/node-v10.15.3-linux-x64/bin

Since I didn’t use apt-get I had to append the node bin directory to my path.

npm version check

user@go-devel ~> npm -v
6.4.1

dremio-oss

Next we’ll clone the dremio-oss package from github.

Clone

Pull the code

git clone https://github.com/dremio/dremio-oss.git

build

Start the build.

cd dremio-oss
mvn -DskipTests clean package

Expect to wait a while while all the various dependency packages are downloaded into your image.

Overall, the first build took about 40 minutes. Unfortunately, I didn’t get a clean build time because I had to stop it in the middle to push everything under screen.

[INFO] Distribution - Server Packaging .................... SUCCESS
[02:28 min]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 20:41 min
[INFO] Finished at: 2019-03-09T15:03:30+00:00
[INFO] Final Memory: 282M/698M
[INFO] ------------------------------------------------------------------------

Here is the cpu graph for the gcp instance: Dremio-build-cpu

base permissions

Reading through the sparse documentation, I’ll try setting base username and password:

mvn compile exec:exec -pl dac/daemon

Not sure if this works… doesn’t appear to.

[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 11.176 s
[INFO] Finished at: 2019-03-09T15:13:47+00:00
[INFO] Final Memory: 51M/123M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal on project dremio-dac-daemon: Could not
resolve dependencies for project
com.dremio:dremio-dac-daemon:jar:3.1.6-201903070042400578-fdcd3a8:
Failed to collect dependencies at
com.dremio.extras.sabot:dremio-extra-sabot-kernel:jar:3.1.6-201903070042400578-fdcd3a8:
Failed to read artifact descriptor for
com.dremio.extras.sabot:dremio-extra-sabot-kernel:jar:3.1.6-201903070042400578-fdcd3a8:
Could not find artifact
com.dremio:dremio-root:pom:3.1.6-201903070042400578-fdcd3a8 in
dremio-free (http://maven.dremio.com/free/) -> [Help 1]

fire up.

OK, fine, let’s see if we can fire this up using the described approach.

I’m not sure what networking hoops I’ll have to play on GCP to get this visible to me. I’m not going to leave it up long as I don’t want others to try and access.

user@go-devel ~/s/j/dremio-oss>
distribution/server/target/dremio-community-3.1.6-201903070042400578-fdcd3a8/dremio-community-3.1.6-201903070042400578-fdcd3a8/bin/dremio start

user@go-devel ~/s/j/dremio-oss> ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
user      4000  3999  0 14:42 pts/1    00:00:00 -/usr/bin/fish
user      5344     1 94 15:16 pts/1    00:00:24 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.util.logging.config.class=org.slf4j
user      5576  4000  0 15:17 pts/1    00:00:00 ps -f

curl tells me that from the gcp instance, a local connection works…

curl http://localhost:9047

However from a browser on my local side, no luck. Connection refused. So I probably have to open up the port in gcp. It’s probably a good thing that a new compute node comes up with only limited ssh firewall ingress…

Yes I had to setup a firewall ingress rule to allow traffic from my specific ip address into the GCP network in order to connect to the dremio-oss app.

Here is the welcome screen. Dremio-oss-signup

Summary

It may look easy, but it took me several attempts to get to the point that I could get the build repeatable enough to demo. Now on some interesting java enhancements.