Optimal Isn't Optional

Although I don’t know much about statistical testing or website optimization, I do want to. That’s why I’m making a resolution to learn more about it next year. (Please suggest your favorite resources in the comments.)

But, since we’re both here, let’s talk about Flask-MAB: an implementation of the multi-armed bandit testing protocol and an alternative to A/B testing.

Written by Mark Grey, Flask-MAB allows you to test different copy, colors, and other UI elements in your app. And since it’s a Flask extension, it bolts right onto your existing Flask app.

Depending on the type of bandit model you choose, MAB tests will converge toward an optimal solution, which means fewer lost conversions during long term tests.

In this example, Flask-MAB will help us determine the optimal button color for our app based on user responses, (adapted from the readme):

```python from flask import Flask,render_template from flask.ext.mab.storage import JSONBanditStorage from flask.ext.mab.bandits import EpsilonGreedyBandit from flask.ext.mab import BanditMiddleware,choose_arm,reward_endpt app = Flask(‘app’)

define test parameters & randomize color 30% of the time.

cb = EpsilonGreedyBandit(0.3) cb.add_arm(“green”,”#2ECC40”) cb.add_arm(“red”,”#FF4136”) cb.add_arm(“blue”,”#0074D9”)

setup storage

app.config[‘MAB_STORAGE_ENGINE’] = ‘JSONBanditStorage’ app.config[‘MAB_STORAGE_OPTS’] = (‘./mab.json’,)

initialize and attach middleware to app

mab = BanditMiddleware() mab.init_app(app)

app.add_bandit(‘btn’,color_bandit)

main route which passes bandit parameter to main.html template

@app.route(“/”) @choose_arm(“cb”) def home(cb): return render_template(“main.html”,color=cb)

route rewards a bandit parameter for succesfull conversion.

@app.route(“/converted”) @reward_endpt(“cb”,1.0) def reward(): return render_template(“thanks.html”)

if name == ‘main’: app.debug = True app.run() ```

After futzing around with Flask-MAB for an afternoon, it seems like a great tool for beginners like me. It’s easy to setup for small projects, it saves data to a single json file, and it can test almost anything you can pass to the UI.

That said, before you use MAB, A/B, or any type of statistical testing in production, you must read this, this, this, this, this, and this.

Do you even comment bruh?
2017 Neal Shyam