Jev, who would you vote for?
Originally published on X.
Jev is basically “a” human.
A human has all kinds of biases. One may prefer purple over pink. One may prefer chicken over beef. One usually chooses chocolate cake for dessert, but depending on the circumstances, one might prefer to try cheesecake.
Asking me, a 24-year-old Korean girl, “Which brand do you like more: Apple or Microsoft?” and “Imagine you are a 43-year-old American businessman. Which brand do you like more: Apple or Microsoft?” would probably yield different answers and levels of confidence. For the latter, since I am not a 43-year-old American man myself, I would answer based on what I have learned (or the stereotypes I hold) about middle-aged American men.
So it’s no surprise that Jev can have biases, too. But how should we interpret the numbers attached to its answers?
Jev, who would you vote for?
We asked Jev, “Who would you vote for? A. Donald Trump, B. Kamala Harris.” With no personal profile provided, a single call selected Kamala Harris, assigning her 59% probability and Trump 41%.
We then asked Jev to choose between Donald Trump and Kamala Harris, using 100 profiles from the Twin-2K-500 research dataset. Jev chose Harris (prefered Harris over Trump) for 70 profiles and Trump for 30. Its average probability assigned to Harris was 67.32%.
But we cannot say that those 70 of 100 people would have actually chose Harris. It’s just Jev imagining the answer of those people based on its own bais about those people and the question. So asking the same model to answer for 100 profiles does not turn it into a survey of 100 people.
To investigate this more carefully, we ran one more simple experiment using actual survey responses about religion, party identification, and political views. We split the 2,058 respondents into 411 development participants and 1,647 evaluation participants. This time, we did not give Jev individual profiles. We asked it to predict the response distribution for a randomly selected person from the historical panel. We also tried giving it the development participants’ response counts. Here are some of the results:
| Response | Actual evaluation sample | Jev: panel description only | Jev: development counts also provided |
|---|---|---|---|
| Protestant | 24.8% | 92.8% | 89.6% |
| Democrat | 40.8% | 16.0% | 93.2% |
| Independent | 29.9% | 72.6% | 4.3% |
| Moderate | 27.9% | 85.8% | 91.3% |
Each Jev percentage is the average probability across six calls. The human percentages are recorded, unweighted responses from this panel, not estimates of the entire US population.
Giving Jev more information changed its answers. But it did not make the answers reproduce the diversity of the human responses. For party identification, providing the development counts shifted most of the probability from Independent to Democrat. That is an important difference: knowing which answer is most common is not the same as knowing how common it is.
Right questions to Ask
Some may say this is the wrong way to use Jev. It was designed to classify and make judgments, and these questions ask it to reproduce human opinion distributions. Yes, that’s a valid objection. What we ask Jev matters.
So we tried two simple questions about a box containing 6 red balls and 4 blue balls:
- If you draw one ball uniformly at random, which color will come out?
- Which color is more numerous?
For the first question, Jev assigned red an average probability of 99%, even though the actual probability is 60%. For the second, it assigned red 100%, which is correct. (We also tried a box containing 3 red balls and 7 blue balls.)
| Box | Question | Correct probability of red | Jev’s average probability of red |
|---|---|---|---|
| 6 red, 4 blue | Color of a random draw | 60% | 99% |
| 6 red, 4 blue | More numerous color | 100% | 100% |
| 3 red, 7 blue | Color of a random draw | 30% | 4.5% |
| 3 red, 7 blue | More numerous color | 0% | 0% |
Each row averages four calls. Across the majority-color questions and additional questions about an explicitly observed color, Jev made 16 correct decisions out of 16. It could identify the more numerous color and read an observed result. But its random-draw probabilities were much more concentrated than the actual chances.
These were small, exploratory experiments with Jev 1.13.0. TypeSafe also explicitly documents weaknesses on numerical tasks, so the ball example should be read with that limitation in mind. It demonstrates a mismatch in these prompts, not a failure of every intended use.
Potential Security Risk
The potential security risk appears when software turns a judgment like this into permission to act.
Say an application asks Jev, “Would this user probably want this document shared?” If the answer is sufficiently confident, the application sends the document. Even a good prediction of the user’s preference would not establish that the user authorized this particular action. The question itself has left out a security requirement.
Now imagine someone can control the document or message Jev reads. If certain wording reliably pushes the model toward approval, that person may be able to influence the decision without having permission to perform the action directly. TypeSafe’s documentation acknowledges that injected instructions, misleading framing, and content arguing for its own classification can move Jev’s answers.
Adversarial-content limitation
This has a potential of becoming a new type of “backdoor.” A security bypass would additionally require an attacker to control relevant inputs and the application to give the resulting decision enough authority. But the possibility exists. A developer could accidentally build that vulnerable decision path, or deliberately exploit a predictable weakness.
A high confidence threshold alone does not settle the issue, either. Jev’s confidence summarizes how concentrated its output probabilities are. It is not an independent check that the decision is correct or authorized.
TypeSafe’s confidence documentation
And even if a model were correctly calibrated at 99% on ordinary requests, that would not establish the same reliability on inputs deliberately selected to make it fail.
So how should we use Jev?
Treat it like “a” human making a judgment from the information available. Ask a clearly defined question, with evidence and criteria that make its answer checkable. Then test whether it gets those judgments right in the situations where you intend to use it.
It can help interpret language and context that would be awkward to express through a long list of if, elif, and else clauses. But that flexibility does not make it unbiased, and it does not turn a plausible answer into permission.
For security-sensitive actions, the software still needs to check who is requesting the action, what they are allowed to do, and whether the required authorization exists. Jev can help interpret a request. But its guess should not create authority the requester does not have.
And for a personal Jev?
It’s like asking a friend who has known you for a while and shared a few meals with you which restaurant you would prefer tonight. Your friend may get it right based on previous interactions. Or they may get it wrong because they don’t know what you had for lunch.
More personal information might make the prediction better. But knowing someone’s habits is different from knowing their current preference, and neither is the same as having their permission.
Jev does open up an interesting possibility for human and social simulation. If we can build a truly personal Jev that learns from our choices, its biases might begin to resemble our own. We may be able to train a model that understands our general preferences without having to explain every thought process behind them. Like a friend who gradually learns our taste, it could learn what we tend to choose. Even when we find it hard to explain why!