Add dockerfile and update README

[mpardesh]

No description provided.

[michael-mclawhorn]

You might want to use ${BRANCH} instead of $BRANCH.

[kevinhinterlong]

We probably want to move the stuff you added here into a separate section because the wikipedia data in the docker conatiner is different from the wikipedia data in the druid quickstart tutorial

@michael-mclawhorn ?

EDIT: actually since this is embedding the generic example around wikipedia data we could move this into fili-wikipedia-example (still leaving a comment that the two datasets are different.)

[mpardesh]

@kevinhinterlong Thank you for your comments! I updated the commands and added some links, and I pushed the (fixed) image to Dockerhub. I also moved the Docker instructions from the generic example to the wikipedia example. Please let me know if it's still not working, or if there's anything else I should change. I am just working on a bit of formatting in the readme now.

[kevinhinterlong]

Pretty good, I'm not sure what we'll end up doing about actually deploying the files to docker though

Maybe make a fili account? @michael-mclawhorn

[kevinhinterlong]

I got an error running the image from docker hub complaining that git wasn't available while fili was trying to start up. So we should probably keep git around

ERROR] Exception while executing SCM command. Error while executing command. Error while executing process. Cannot run program "git" (in directory "/usr/local/fili/fili-generic-example"): error=2, No such file or directory

[kevinhinterlong]

Should probably add a link to the page on docker hub.

Also maybe add a link to the docker install page

[kevinhinterlong]

These were the commands I had to run to start it up

docker pull mpardesh/fili
docker run --name fili-generic-example -i --rm -p 3001:8081 -p 3000:8082 mpardesh/fili:1.0

also there should probably be a -p 9998:9998 so that people can connect to fili.

[mpardesh]

Thanks for the suggestions. I have added those commands into the readme. I also thought that the 9998 mapping was necessary, but when I tried to run with -p 9998:9998 Docker gave me an error:

Error response from daemon: driver failed programming external connectivity on endpoint fili-wikipedia-example
Error starting userland proxy: Bind for 0.0.0.0:9998 failed: port is already allocated.

[kevinhinterlong]

So it looks like fili is starting up too quick (at least on my machine), so we should wait until druid has finished setting up before letting fili configure itself.

[kevinhinterlong]

Should probably add the version tag here as well mpardesh/fili:1.0

[kevinhinterlong]

Ok so I just ran it and it started up correctly, but unfortunately fili-generic-example started up too quickly and druid wasn't ready so it didn't detect any tables. You might have to write a startup.sh as your ENTRYPOINT and run all the commands there.

Then you could put something like

echo "Waiting for Druid to finish setting up"
while ! curl http://localhost:8081/druid/coordinator/v1/datasources | grep -q "wikipedia"; do
  sleep 5
done
echo "Druid finished setting up. Starting Fili"

before running the generic example so that it's forced to wait until druid is ready.

[mpardesh]

Sounds good. I'm working on the shell script right now. Quick question, since we mapped host port 3001 to container port 8081 should we use port 3001 in the while loop instead of 8081? I tried going to
http://localhost:8081/druid/coordinator/v1/datasources
while the container was running but I couldn't connect to port 8081. It seemed to work when I did this instead:
http://localhost:3001/druid/coordinator/v1/datasources

[kevinhinterlong]

Since the script is running inside the container it should only be able to see port 8081 (port 3001 is visible from outside the container which is why you could connect to that but not 8081). So it should connect on port 8081 inside the script.

[kevinhinterlong]

In response to your earlier comment it looks like you must have already had something using that port.

The first part of the port argument is the host port, so you could always do something like -p 9996:9998, and then check the here to make sure it connected, but it should forward some port to the docker container.

[kevinhinterlong]

I think this should be docker stop fili-wikipedia-example

[kevinhinterlong]

This shouldn't be here since we're cloning the repository in although you will need to changeGITHUB_OWNER=mpardesh and BRANCH=filiDocker while working on it locally (i.e. don't commit that change).

That way when filiDocker gets merged into master everything should work the same.

You might still have to run chmod and you'll need to update the path to entrypoint.sh for the chmod and ENTRYPOINT

[mpardesh]

I updated the Dockerfile and entrypoint.sh and uploaded the new image to Dockerhub to test. It seems to work on my computer, but then I tried pulling and running the image from a different computer and got this message when I sent the query:

{
    "status" : 400,
    "statusName" : "Bad Request", 
    "reason" : "com.yahoo.bard.webservice.web.BadApiRequestException",
    "description" : "Table name 'wikipedia' does not exist.",
    "druidQuery" : null 
}

I'm not sure what's wrong.

[kevinhinterlong]

Before you were backgrounding this process and I'm guessing you want to do the same here

/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf &

It looks like it waits here and never tries to start up fili

[michael-mclawhorn]

@kevinhinterlong did you change to 'needs 1 reviewer' because you were done reviewing? If so, you still need to accept changes.

[kevinhinterlong]

@michael-mclawhorn I'm pretty confident this will work once the last comment I made is implemented.
The only other things I can think of is stuff you might have to decide on like if we should republish the file on docker hub under a fili account.

[kevinhinterlong]

Small fix

[kevinhinterlong]

This should stay as master

[kevinhinterlong]

Another smallfix: Looks like curl is somehow not installed

/usr/local/fili/fili-wikipedia-example/entrypoint.sh: 6: /usr/local/fili/fili-wikipedia-example/entrypoint.sh: curl: not found

EDIT: Ok I just tried it locally after adding curl and it worked so this should be the last fix.

[mpardesh]

I could add

RUN apt-get install -y curl

before the chmod in the Dockerfile. Is there a better way to do this?

[kevinhinterlong]

I was thinking just change the current line to

RUN apt-get update && apt-get install -y --no-install-recommends git curl

[mpardesh]

@kevinhinterlong After I added curl to the end of this line, Codacy had some complaints, like

• either use wget or curl but not both

• specify a version when you use apt get install

• delete the apt-get lists after installing something

Should I try to fix these issues?

Codacy also complained about some things that I already fixed, like not specifying a maintainer. Strangely, it didn't have any issues with the previous commit.

[kevinhinterlong]

Builds fine for me 👍 also we ended up at 1.35GiB RAM used while running which is nice.
@michael-mclawhorn I think the biggest thing left would be deciding about hosting it on docker hub.

[archolewa]

Is this necessary? Or did this slip in because we haven't rebased this branch recently?

I ask mostly because I remember Mike saying we had an issue with Travis and the ssh-agent running continuously, and I want to make sure we don't regress once this gets merged in.

[kevinhinterlong]

This got slipped in from Mike's PR

[archolewa]

I'm not going to block on this, but we may want to add a timeout here, so that docker doesn't just hang if Druid is having a problem loading up.

[archolewa]

We can remove this line. The entrypoint script ensures that Druid will be fully up and running by the time the image is up and running.

[kevinhinterlong]

While it's true that druid is up and running it still has to index the data which can take some time (I believe the druid Dockerfile this is based on is fetching some data from S3 as well)

[archolewa]

@kevinhinterlong That was my thought. We should probably create a fili-io user on the docker hub and upload the dockerfile there.

[kevinhinterlong]

@michael-mclawhorn
Ok so with regards to automated builds, it looks like we have 2 options

1. Use automated builds from Docker Hub
2. Have travis builds which trigger ci-tag.bash also login, build, and push a new build to DockerHub.

The first is definitely easier and probably the way to go, but it looks like we could also do some combination of the two using Remote build triggers on Docker Hub.

The second option would look something like

# ... appended to ci-tag.bash

#Login to docker
docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS

#Build the dockerfile
docker build -t fili-wikipedia-example -f fili-wikipedia-example/Dockerfile .

#only push a successful build
DOCKER_BUILD_RETURN_CODE=$?
if [[ ${DOCKER_BUILD_RETURN_CODE} -ne 0 ]]; then
    echo "ERROR Docker build did not succeed."
    exit ${DOCKER_BUILD_RETURN_CODE}
fi

#tag and push the new build
REPO=fili/fili-wikipedia-example
docker tag fili-wikipedia-example $REPO:$NEW_TAG
docker push $REPO

[kevinhinterlong]

Ok we definitely want to go with the first option. I just played around with it and it should be much easier + it's independent of travis-ci so as long as the Dockerfile is fine it'll build fine (which we always expect to be the case on master)

[archolewa]

@QubitPi Can you verify that changes have been made to your satisfaction? @mpardesh is going back to school this weekend, so I want to get her stuff merged in nowish.

Since it has two approvals on it, and I don't want to leave this hanging in perpetuity, I'm going to discard your review and merge it tomorrow if you don't take a second look by then.

@mpardesh Please rebase this on top of master, so we can merge it in.

[mpardesh]

@archolewa thanks! I have rebased this

[archolewa]

I've migrated this branch off of @mpardesh's fork and onto the main fork so that we can continue to rebase it and keep it up to date until we get a chance to create a fili dockerhub. See #523 for the new PR.