About feature flag allocation

When does a visitor enter a test, and what decides the group they land in? The answer is not "when you create the feature flag" and not "when the page loads". It is the moment something reads the bucket, and that moment is yours to choose.

A registration is not a decision

The SDK loads every feature flag in your project at startup. It allocates nobody then. A feature flag stays undecided until something reads its bucket. The first read runs the decide function, stores the result, and reports the decision.

You therefore choose when Picked commits a visitor to a feature flag. You choose it by the point at which you read the feature flag.

A read of the bucket records an exposure

The first read is the decision, so it is also the moment the visitor enters the test. If you read every feature flag at startup, every visitor counts as exposed to every test. That includes a test the visitor never reached.

Read a feature flag where the decision matters: after the cart loads, after the quote returns, or after you know who the visitor is. A feature flag that you read before that point puts a visitor in a test they never saw, and the uplift number then means nothing.

A condition reads the bucket, so a condition allocates

Most reads are not calls that you write. They come from a condition. A condition that points at a feature flag with invoke: { var: 'features.<id>' } must have the bucket before it can compare it. That read allocates the visitor, exactly as a call in your own code does.

Where the condition sits therefore decides who enters the test:

A precondition that points at a feature flag allocates every visitor on your site, and it does so before the visitor sees anything. Keep a feature flag out of a precondition unless it truly controls whether Picked starts at all.

Keep each decide function fast for the same reason. Everything that waits for the condition waits for decide, and that includes the experience itself. A network call inside decide delays what the visitor sees.

How Picked chooses

For a feature flag you control from the dashboard the choice is deterministic. Picked gives each visitor a stable value for each feature flag. It derives that value from the visitor ID and from the ID of the feature flag, and it compares the value against the weight. Picked allocates the visitor when the value is below the weight.

The same visitor gets the same result for one feature flag on every visit, while the weight stays the same.

That is also why a feature flag you control from the dashboard cannot use the data of your application in its decision. The value comes from the visitor ID, the feature flag ID, and the weight, and from nothing else. Define the feature flag in SDK code when the decision needs a cart, a quote, or a profile.

More than two groups

A Text feature flag has a table of variants, and each variant is one group. Picked allocates a visitor to exactly one of them, and the control group is one of the rows.

The two types decide by different routes, and the difference matters if you compare them:

A Text feature flag gives each variant a score. Picked makes the score from the visitor ID, the feature flag ID, and the key of the variant. The weight of the variant then scales that score. The variant with the best score receives the visitor.

The weight of a variant therefore changes the score of that variant alone. No other variant moves because of it.

A Boolean feature flag makes the same comparison it always has. A feature flag that you made before variants existed therefore keeps every visitor where it put them.

Picked derives the stable value from the visitor ID and from the ID of the feature flag, for both types. The two types are therefore independent of each other in the same way that two feature flags are.

The order of the rows has no effect. Picked scores each variant from its key, and the key of a variant never changes. Move a row in the dashboard and the visitors stay where they are. The order that you see is the order that you chose to read them in, and nothing more.

Rename a variant and nothing changes. The name travels no further than the dashboard.

Change the string that a variant serves and the visitors stay in the same group, but your site and your reports receive the new string. A visitor with a decision in browser storage keeps the old string until you increment the version.

Each feature flag is independent

Picked builds that value from the visitor ID together with the ID of the feature flag. Each feature flag therefore gets its own value, and tests that run at the same time are independent of each other.

Two feature flags, both at weight 50, split your visitors into four equal parts: allocated to both, to one, to the other, or to neither. Three feature flags at weight 50 give eight groups of 12.5%. Add a feature flag and the groups divide again.

This is what lets you read two experiments separately. Give every feature flag the same value, and two feature flags at weight 50 hold exactly the same visitors. Your analytics then never sees the mixed groups that tell one result from the other.

The allocation ratchets

The ratchet applies to both types of feature flag. Inside one feature flag, a larger weight only adds visitors. Everybody allocated at weight 30 is still allocated at weight 50. The allocation is a ratchet: it turns one way, and a larger weight never moves a visitor back out.

That property is what makes a staged release safe. A visitor who sees the change at 10% still sees it at 20%. A smaller weight is the reverse turn. It removes visitors from the top of the range, and every visitor below the new weight keeps what they had.

A Text feature flag has the same ratchet, one variant at a time. A larger weight on one variant only adds visitors to that variant. A smaller weight only removes them. Two variants that you do not change never exchange visitors with each other.

A disabled feature flag records nothing

A disabled feature flag allocates nobody, and the difference from a weight of 0 matters for your reports. At weight 0 the visitor is in the test, and in the control group. When the feature flag is disabled, the visitor is not in the test at all.

The SDK does not evaluate a disabled feature flag. It writes no decision to browser storage and it reports no decision. The feature flag is also absent from the feature list on every analytics event. Expect a step change in your reports when you disable one: fewer feature flags for each visitor, and fewer decision events.

This state is temporary. A visitor allocated before you disabled the feature flag keeps that bucket in the browser. When you enable the feature flag again, that visitor continues in the same group.

What a version does, and what it does not

Each feature flag carries a version, and it starts at 1. The SDK caches the bucket in browser storage under a key that holds the feature flag ID and the version. A visitor therefore keeps the same bucket across page loads.

Change a weight and leave the version as it is, and that cache decides the result. A visitor who has a cached bucket keeps it. The new weight reaches only a visitor who has no cache. Your reports then hold two groups of visitors at the same time, one measured against the old weight and one against the new.

Increment the version and the old cache key no longer matches. The feature flag counts as undecided, decide runs again on the next page load, and every visitor is measured against the new weight. Increment it after each weight change, unless you intend the change to reach new visitors only.

The version affects the cache only. It is not part of the allocation. The ratchet still holds after you increment the version: a visitor allocated at weight 30 is allocated again at weight 50. You cannot decrement a version.

When the SDK reports a decision

The SDK sends feature_decide the first time that it allocates a visitor to a bucket, and only then. On a later page load it restores that bucket from browser storage and sends nothing. A visitor who returns therefore produces no feature_decide event, although that visitor is still in the group they went into the first time.

An integration that listens only for feature_decide under-reports every visitor who returns. Choose the source that matches your question:

listStore updates each time that a feature flag resolves. It does this whether the SDK restored the bucket from storage or allocated the visitor now. A watch on it therefore always gives the current state. Use feature_decide only when you need to know that Picked allocated a visitor at that moment.

What Picked reports for you

Every Picked analytics event carries a feature_list context, and a report of your own can use it. The context describes each feature flag that this visitor resolved: the feature flag ID, the name, the version key, and the bucket. Only a resolved feature flag appears there, so the context shows what the visitor was exposed to at the time when the event fired.

The optional Sentry plugin also tags each allocation onto the Sentry scope as picked.feature.<id>, with the bucket as the value. You can then filter your errors by the variant.

Preview mode

In preview mode the SDK stores the bucket in session storage and not in local storage. The bucket does not stay after you close the tab, and Picked can allocate a preview visitor differently from a production visitor. Treat a preview allocation as disposable.

A feature flag that you define in SDK code can ask for the same thing in production with storage: 'session'. Use it for a short test that must not last across browser sessions.

Related

• How to read a feature flag — the accessors, and which ones resolve the feature flag.
• Feature flags — the dashboard controls.
• SDK feature flags — the definition shape and the accessor table.