Intuit Wasabi Setup Hell

Wasabi is an open-source A/B Experiment Server by the developers at Intuit.  If you are currently pulling your hair out trying to set this thing up, R-E-L-A-X.  

First off:  It's not your fault.  The developers dumped this shit onto Github and had a good laugh.  At your expense.  

Secondly: Suckatash is here to walk you through my week of hell. Take the journey with me and you may be rewarded.  Or not.  Either way, you got what you paid for.  It's open-source after all!

---

The project can be found here:

https://github.com/intuit/wasabi

The first thing you should do is ignore the self-install instructions. They heavily rely on Docker. It won't work.  It will create three docker containers.  One for the Wasabi Java-based server.  One for a vanilla MySQL install.  And one for Cassandra.  You are better off setting up those systems discretely on your favorite cloud-based service.  I'm an avid AWS user.  So that's what I did.

Let's start with setting up a DYI Cassandra cluster....

---

STEP ONE: INSTALL CASSANDRA CLUSTER ON EC2 (three instances)

The following steps need to be executed on each instance.  The three instances should be created on the same subnet so they can talk to each other:

$ sudo echo "deb http://www.apache.org/dist/cassandra/debian 311x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
$ sudo curl https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add -
$ sudo apt update
$ sudo apt install cassandra 
$ sudo service cassandra stop

Alter the security group on all three instances to allow each other to talk to the following ports:

• 7000
• 7001
• 7199
• 9042

For reference, here is what my security group for the cassandra instances look like:

Ports Protocol Source

7199 tcp 172.31.0.0/16

7001 tcp 172.31.0.0/16

22 tcp 0.0.0.0/0

7000 tcp 172.31.0.0/16

9042 tcp 172.31.0.0/16

With all three instances stopped, do the following:

$ sudo rm -rf /var/lib/cassandra/data/system/*

$ sudo vi /etc/cassandra/cassandra.yaml

And set values to the following inside this YAML file:

cluster_name: 'Wasabi Cluster'

authenticator: AllowAllAuthenticator

seeds: "172.31.17.203,172.31.20.247,172.31.27.209"

listen_address:    (yes, this is blank!)

rpc_address: 0.0.0.0

broadcast_rpc_address: 172.31.17.203 (the address of the host you are on)

endpoint_snitch: GossipingPropertyFileSnitch

The ip addresses are private ones issued to me. Yours will be different. Once the yaml file is set, do the following:

$ sudo service cassandra start

$ sudo tail -f  /var/log/cassandra/system.log

Once the service has finished starting, check on the status of each node to see if they are discovering each other:

$ sudo nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID    Rack
UN  172.31.17.203  717.19 KiB  256          30.3%             XXXXXXX   rack1
UN  172.31.20.247  331.06 KiB  256          36.3%             XXXXXXX   rack1
UN  172.31.27.209  741.18 KiB  256          33.4%             XXXXXXX   rack1

Once your node status looks something like mine above, you're done with setting up a Cassandra Cluster.

---

STEP TWO: Use Wasabi to seed data on your  CASSANDRA CLUSTER

It is very important that you spin up an UBUNTU 16.4 server.  No other version of UBUNTU (or other linux variant) is supported.  I have no idea why.  I just know that I tried initially with UBUNTU 18.x and nothing worked.  

Make sure you open up these inbound ports on your EC2 instance security group.  You'll need all of these later:

9000
35729
8080
22

Once your instance is up, login and start with the following commands:

$ cd /home/ubuntu
$ git clone https://github.com/intuit/wasabi.git
$ wget https://oss.sonatype.org/content/repositories/public/com/builtamont/cassandra-migration/0.9/cassandra-migration-0.9-jar-with-dependencies.jar

$ export CASSANDRA_MIGRATION=/home/ubuntu/cassandra-migration-0.9-jar-with-dependencies.jar
$ export MIGRATION_SCRIPT=/home/ubuntu/wasabi/modules/repository-datastax/src/main/resources/com/intuit/wasabi/repository/impl/cassandra/migration


For the CQLSH_HOST, use one of the three ip addresses used to set up your cluster:

$ cd ./wasabi

$ CQLSH_VERSION=3.4.4 CQLSH_USERNAME= CQLSH_PASSWORD= CQLSH_HOST=172.31.17.203 ./bin/docker/migration.sh

Your Cassandra cluster should now be seeded with the data required by Wasabi.

---

STEP THREE: Wasabi Server install and setup

$ cd /home/ubuntu/wasabi

$ git checkout 1d2f066541b176ee84c00dc9516b370553b76a40

$ ./bin/wasabi.sh bootstrap

$ ./bin/wasabi.sh -t false package

$ sudo dpkg -i ./target/wasabi-main-build_1.0.20180226051442-20181025080918_all.deb

This should install wasabi under the directory:  '/usr/local'

---

STEP FOUR: Install MySQL Server locally

Make sure your "root" user is set with a wildcard ('%') and not 'localhost'!!!  Also notice no password.

Add a 'wasabi' user:

mysql> create user 'wasabi'@'%' identified by '';
mysql> grant all privileges on *.* to 'wasabi'@'%' with grant option;

mysql> flush privileges;

Seed the MySQL Server with the following schema:

https://s3-us-west-2.amazonaws.com/gardella.org/wasabi_mysql_dump.sql

$ mysql -u root -p < wasabi_mysql_dump.sql

---

STEP FIVE: Start the Wasabi Server

  WASABI_CONFIGURATION="

  -Ddatabase.user=root\

  -Ddatabase.password=<your mysql root password>\

  -Dusername=\

  -Dpassword=\

  -DnodeHosts=172.31.17.203,172.31.20.247,172.31.27.209\

  -DtokenAwareLoadBalancingLocalDC=dc1\

  -Dapplication.http.port=8080" bash /usr/local/wasabi-main-1.0.20180226051442-build/bin/run &

Tail the wasabi server console log file.  Mine was found here:

/usr/local/wasabi-main-1.0.20180226051442-build/logs/wasabi-main-1.0.20180226051442-build-console.log

It should not get stuck reading the mysql database.  It should go fairly quickly.  Within 60 seconds it should stop with the following line:

[HttpService STARTING] INFO  com.intuit.autumn.web.HttpService - started HttpService  

---

STEP SIX: Start the Wasabi Front-End server

The front end service is a separate node application.  You can run both the server and front-end UI on the same EC2 box.  Run the following:

$ cd /home/ubuntu/wasabi/modules/ui

$ npm install

$ bower install

$ grunt build

If all goes well (and more than often, it does not...) edit the following Gruntfile:

$ vi wasabi/modules/ui/Gruntfile.js :

development: {
    constants: {
    supportEmail: process.env.SUPPORT_EMAIL || 'you@example.com',
        apiHostBaseUrlValue: process.env.API_HOST || 'http://<your_domain_here>:8080/api/v1',
        downloadBaseUrlValue: process.env.API_HOST || 'http://<your_domain_here>:8080/api/v1'
    }
}


$ vi wasabi/modules/ui/default_constants.json :

    "apiHostBaseUrlValue": "http://<your_domain_here>:8080/api/v1",
    "downloadBaseUrlValue": "http://<your_domain_here>:8080/api/v1"


Now you should be ready to start the UI service:

$ grunt serve:dist &



To see the login screen:

http://<your_domain_here>:9000/

The default admin account works out of the box with password: admin


I was not able to create additional users.  The implementation is file based and even altering that file, rebuilding the rpm and deploying it had no effect.



If you want to see the swagger API documentation (very helpful) you can find it here:

http://<your_domain_here>:8080/swagger/index.html#/

How to get a custom font into an AWS Lambda

I had a problem.  For the longest time, I've been able to avoid localization.  Back in 2008, I shipped Spore with 22 different languages.  You haven't lived until you deal with kerning issues while shipping Thai inside a video game UI.

Enter my most recent year with Viber.  A company founded in Tel Aviv and owned by a Japanese multinational, it's easily the most international company I've ever worked for.  Which brings me to my latest technical challenge:  

Russia

I don't remember localizing Cyrillic before.  And the wrinkle I needed to deal with is that fact that in 2013, the currency symbol changed to ₽.  It was adopted in 2014 by the Unicode Technical Committee and released as part of Unicode 7.0 in June of that year. Unicode nerds rejoiced! 

Do you have any idea how often default font sets get updated in standard cloud-based linux server images?  Neither do I, but I'm guessing not often enough.  I can tell you that Unicode 7.0 is not available on any default Amazon Linux image I've found.  Which brings me to my next part of the mine field: Lambdas

What you need to know about Lambdas is that you as a developer don't really get to control the whole server instance in the cloud.  You get a tiny bit of developer space to drop your code into.  How the heck am I going to install an additional font set onto a Lambda?

Enter fontconfig.

Bless you, fontconfig! It makes this kind of thing possible.  I owe my success with this week-long odyssey to the makers and maintainers of it.  fontconfig is a library that allows you to do exactly what its name implies: 

Configure where the operating system goes to look for fonts

If you've ever printed a document and you start seeing these funny question marks all over the place, you know the kind of fun I'm talking about:  

� � � � �

Which is sort-of-exactly what I was getting from my image-compositing program built with Node and running on a Lambda:

And with a little magic with fontconfig, I could generate it correctly:

Amazing!  How did I pull this off?  Are you having this problem too?  Let me outline my journey for you:

Step One: Go Down a Blind Alley

A helpful coworker gave me this link.  It showed promise.  This person was trying to get CJK (Chinese, Japanese, Korean) characters to render inside a headless chrome instance running inside an AWS Lambda.  I only sorta know what he/she is talking about.  But I see LAMBDA and FONT and CHINESE CHARACTERS. So I know we are in the ballpark.

I installed docker so I could run inside a container that closely emulates the environment that Lambda runs in.  I download, configure, make and make install fontconfig from source. Ugh. I find a google font set that contains my missing cyrillic ruble currency symbol. (hey look: Bitcoin made it too!)

It turns out that I needed to do almost none of this.  Well, I did need to download the True Type Font set "Roboto" from Google.  But I digress.

Step Two: Try The Simple Way

My lambda code is written in Node. I figured that when I zipped up my existing code to upload to AWS Lambda with my additional special font, it would need to look something like this:

/my_lambda_code
    index.js
    /node_modules
    /roboto  (my new font folder)

But how do I tell me Node program to use this font?  I was at a loss until this Stack Overflow answer gave me what I needed.

What's important is what is INSIDE the "/roboto" subdirectory:

/roboto
    Roboto-Black.ttf
    Roboto-BlackItalic.ttf
    Roboto-Bold.ttf
    Roboto-BoldItalic.ttf
    Roboto-Italic.ttf
    Roboto-Light.ttf
    Roboto-LightItalic.ttf
    Roboto-Medium.ttf
    Roboto-MediumItalic.ttf
    Roboto-Regular.ttf
    Roboto-Thin.ttf
    Roboto-ThinItalic.ttf
    fonts.conf

The fonts.conf file looks like this:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <dir>./roboto/</dir>
  <cachedir>/tmp/fonts-cache/</cachedir>
  <config></config>
</fontconfig>

This tells fontconfig that "roboto" is the only font available.  Use it no matter what.  The only step missing is how to tell the Node runtime to use this config file. Once again the Stack Overflow user Jeremy Green had my back.  All I needed was one environment setting in my Node script:

process.env.FONTCONFIG_PATH='./roboto'

Boom.  I'm a Unicode 7.0 rendering madman.

Step Three:  I lied about how easy this was

I really left a bunch of clues out.  Here is a Github repo that adds SVG support to the existing ImageMagik installed on an AWS Lambda Linux VM.  It uses the same fontconfig trick.  That got me on the right path.  Installing a bunch of packages and compiling from source on a docker image was useful for learning about these underlying technologies but ultimately unnecessary for solving my problem.  Most of my professional life in software is like this.  Wandering down paths I have little expertise in.  Making huge time wasting mistakes.  This journey was fun and rewarding.  Which was so unusual I felt compelled to blog about it.  Thanks for reading.

Join Our Hive Mind

This is the amazing form.  

Thanks for signing up!

Look at my customs!

Email Address

Company

Birthday /

Anniversary / /

It's your Mom's Birthday / /

Say something nice to me

Email Lists

General Interest

test list

test list 2

By submitting this form, you are granting: EFF ONE, 1234 Foo Ln, Orinda, California, 95050, United States, permission to email you. You may unsubscribe via the link found at the bottom of every email. (See our Email Privacy Policy for details.) Emails are serviced by Constant Contact.

Spring Boot and JUnit

New gig is ramping up the use of Spring Boot. Absolutely love it. New gig is also committed to the idea of micro-services. There's pros and cons with that one. Moving on... Start with an example like this: 

https://spring.io/guides/gs/accessing-data-rest/ 

And we can temporarily store some rest/json data in a quick spring boot app with a quick maven command: 

mvn spring-boot:run

The trick is that if you want to run a j-unit test against this new web service, how do you identify it?  Should it spawn a new thread?  Should the test have a static dependency somewhere else in the test harness?

With a soup of annotations, you get the right way to do this unit test:

http://www.jayway.com/2014/07/04/integration-testing-a-spring-boot-application

But here's the thing about the above blog post.  It won't work.

It's missing a critical static pair of methods:

private static ConfigurableApplicationContext appContext;

    

@BeforeClass

public static void startBootApp(){

    appContext = SpringApplication.run(MockDataApplication.class, "");

}

    

@AfterClass

public static void shutdownBootApp(){

    appContext.close();

}


You have to do execute these statically.  You only want one spring boot application running before the unit test runs.  If you use @Before annotation, you'll get a new app launched before every @Test block.  

So that's it.  It's pretty sweet.  You load up the JPA repository with data, then you run tests against that web service, checking for the data.  Plus the RestAsssured library has a very slick json validation format. 

Work in progress of a test project can be found here:

https://github.com/bgardella/spring-boot-example

Spring Async and Future

I had a pesky reporting job that was going to be slow no matter what sql-fu I was going to come up with.  I knew that Spring had asynchronous tasks which I had used before but they kinda sucked.  Lots of manual bean building and xml config nasties.  But behold, I came across a nice annotation I was not aware of:

@Async


And I was off to the races.

I'll update with my example later.  For now, I give you the best blog entry to be found as of yet:

http://blog.inflinx.com/2012/09/09/spring-async-and-future-report-generation-example/

Thanks, Random Thoughts.

AVPlayer audio problems?

I'm finally working on an iphone app.  Yes, welcome to 2008!  And this one goes out to anyone who can't get their audio to play on their device, but it will work on simulators!  Fun!

Grrrrrr.

So here's the magic line of objective c that will get your video to play sound on a device.  Are you ready?  Here goes:

[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:NULL];

I'm staring at this line of code.  It appears to say that any errors with audio, ignore them.  So now your audio will work.  Yay, Apple.  This is the future of app developement?

ReCaptcha without plugins

So I'm not that fantastic at reading documentation.  But let's face it.  Coders are not that great at spelling things out.  Exhibit A:

http://code.google.com/apis/recaptcha/docs/verify.html

There is nothing on this API explanation that explains that you can't make this call from javascript.  Although I should have known that putting both public and private keys in JS seems a little insecure.

Ulitimately, you gotta make the reCaptcha verification call from the server-side.  Since I'm a java/spring guy I made a controller that does that.  This is the step that would have been obvious if I just looked more carefully at the example plugins.  But still.

@Controller

public class RecaptchaController {

    public static final String PUBLIC_KEY = "XXX";

    private static final String PRIVATE_KEY = "YYY";

    private static final String RECAPTCHA_URL = "http://www.google.com/recaptcha/api/verify";

   

    @RequestMapping(value="/recaptcha", method=RequestMethod.POST)

    public void checkServer(@RequestParam("remoteip") String remoteip, @RequestParam("challenge") String challenge,

                            @RequestParam("response") String response, HttpServletResponse httpResp){

           

        HttpClient httpClient = new HttpClient();

        PostMethod post = new PostMethod( RECAPTCHA_URL );

        post.addParameter("privatekey", PRIVATE_KEY);

        post.addParameter("remoteip", remoteip);

        post.addParameter("challenge", challenge);

        post.addParameter("response", response);

       

        String resp = "false";

        try {

            httpClient.executeMethod(post);

            resp = getResponseAsString(post);

        } catch (Exception e) {

            resp = "false";

        }

       

        if(resp.startsWith("false")){

            resp = "ERROR";

        }else{

            resp = "SUCCESS";

        }

       

        try {

            OutputStreamWriter writer = new OutputStreamWriter( httpResp.getOutputStream() );

            writer.write(resp);

            writer.close();

        } catch (IOException e) {

            // do nothing

        }

       

    }

   

    private String getResponseAsString(HttpMethod hm) throws IOException {

        InputStream inputStream = hm.getResponseBodyAsStream();

        StringWriter writer = new StringWriter();

        IOUtils.copy(inputStream, writer, "UTF-8");

        return writer.toString();

    }  

}