LABOR-LLM: Language-Based Occupational Representations with Large Language Models

LLMs are conditional probability distributions over text sequences; conditional on a sequence of text (i.e., a prompt), an LLM provides conditional probabilities over all possible continuations of the prompt. Therefore, repurposing LLMs to predict occupations requires representing occupational trajectories as text.

We create a text template, a function that transforms an individual’s career history into a textual summary; this function is denoted by $\mathcal{T}(x_{i},x_{i,\leq t},y_{i,<t})$. Our text template takes advantage of the fact that each occupation has a natural textual representation: its title. For example, the title of the occupation with SOC code 19-1022 is Microbiologists¹¹1Readers can refer to the official Bureau of Labor Statistics (https://www.bls.gov/OES/CURRENT/oes_stru.htm) for a list of the latest SOC titles.. Job titles can be variable in length, depending on how an LLM tokenizes words; for our experiments, the length of job titles ranged from 2 to 28 tokens, with an average length of 8 tokens. (Figure 11 in Appendix D presents a word cloud example of job titles). We use a similar strategy for representing covariates as text; for example, we represent an individual’s educational status using values such as graduate degree.

To elicit an LLM’s predictions of an individual’s next job, we include all previous job information (along with all previous and current covariate information) in a text template. To predict an individual’s $t+1$’st job, the text template will begin with a description of the static covariates and then include a row for each of the $t$ previous occupations and dynamic covariates. It will conclude with a partial row for the occupation to be predicted. For example, the following text template would be used to elicit an LLM’s prediction of an individual’s third job:

<A Resume from the NLSY79 Dataset>The following is the resume of a male white US worker residing in the northcentral region.The worker has the following work experience on the resume, one entry per line, including job code, year, education level, and a description of the job:1988 to 1989 (graduate degree): Secretaries and administrative assistants1989 to 1990 (graduate degree): Carpet, floor, and tile installers and finishers1990 to 1991 (graduate degree):The template omits the title of the individual’s third job (“Elementary and middle school teachers”). When an LLM is prompted with this template, we can record its response as its prediction of the next job.

We can also use the text template to build the individual’s full job history. The example below shows the text representation of a worker’s entire career history generated by our text template. Note that the individual can stay in the same job for multiple records; the text representation explicitly reflects this information. This individual will have five prediction tasks in total, one for each record, throughout their job history. With a slight abuse of notation, let $\mathcal{T}(x_{i},x_{i,\leq T_{i}},y_{i,\leq T_{i}})$ denote the paragraph representing the entire career history of worker $i$.

<A Resume from the NLSY79 Dataset>The following is the resume of a male white US worker residing in the northcentral region.The worker has the following work experience on the resume, one entry per line, including job code, year, education level, and a description of the job:1988 to 1989 (graduate degree): Secretaries and administrative assistants1989 to 1990 (graduate degree): Carpet, floor, and tile installers and finishers1990 to 1991 (graduate degree): Elementary and middle school teachers1991 to 1992 (graduate degree): Elementary and middle school teachers1992 to present (graduate degree): Adult Basic and Secondary Education and Literacy Teachers and Instructors<END OF RESUME>The corpus of text representations of full career histories is useful when fine-tuning language models.

To assess the representativeness of an LLM’s occupational predictions, we compare its predictions to actual occupational trajectories from survey datasets. We study three commonly used survey datasets: the Panel Study of Income Dynamics (PSID) (Johnson et al., 2018) and two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) (Rothstein et al., 2019).

We randomly construct “test samples” containing 20% of individuals in each dataset (test samples contain all observations for each included individual). Table 10 in Appendix C presents summary statistics about each dataset.

For each individual in the test set, we prompt LLMs to predict each recorded observation of their career: predicting their first job from just their covariates, predicting their second job from their first job and covariates, etc. We evaluate a model’s representativeness by comparing its predictions of an individual’s next job to their actual next job. Specifically, we evaluate models by computing their perplexity, a commonly used metric in NLP. The perplexity is a monotonic transformation of log-likelihood, with lower perplexity indicating that a model’s predictions are more representative. Formally, for a model $\hat{P}(y_{i,t}|x_{i},x_{i,\leq t},y_{i,<t})$ that assigns a probability to each possible occupation, perplexity is given by

where $T_{i}$ is the number of observations for individual $i$; $w_{it}$ denotes the sampling weight for the individual; we can adjust these weights to assess the model’s performance on different subpopulations or other objectives. For example, we can set these weights to be such that we weight each transition equally, or we weight each individual equally. We can also set them such that we seek representative predictions only on the first few or last few transitions for every individual; In Section 4 and Section 5, we set $w_{it}=1$ to evaluate models’ performances on the general population. We consider additional evaluation metrics (such as calibration) in Section 6.

While perplexity evaluates the probabilities assigned to occupations, LLMs assign probabilities at the token level. Occupation titles typically span multiple tokens; for example, the title “software engineer” may be tokenized into two tokens, one for “software” and one for “engineer”. However, because LLMs are probabilistic models, we can use the chain rule of probability to extract probabilities assigned to full occupation titles. Equation (3) illustrates how one can obtain the conditional probability assigned to “software engineer”. See Appendix F for more details.

Because evaluating perplexity requires accessing a model’s assigned probabilities, we can only study LLMs whose probabilities are accessible. We study three open-source LLMs from the Llama-2 family of models: Llama-2 (7B), Llama-2 (13B), and Llama-2 (70B); these models were trained on 2 trillion tokens of text from the Internet, and are among the most capable open-source LLMs (Touvron et al., 2023).

When we prompt these LLMs to predict an individual’s future occupations, they provide plausible-sounding trajectories. Readers can refer to Appendix A for examples. However, they also assign mass to strings that are not valid job titles. To encourage models to predict only valid occupations, we consider an additional prompting strategy that includes the list of all possible titles before the prompt.

Table 1 contains the perplexity of each model with both prompting strategies. As a comparison, we also include the perplexity of CAREER (Vafa et al., 2024), a non-language model developed solely to predict nationally representative occupational trajectories. The LLMs consistently make unrepresentative predictions, with perplexities ranging from 39.53 to 3820.31. For comparison, a completely uninformative model that assigns uniform mass to each possible occupation would achieve a perplexity of $|\mathcal{Y}|$, which is 335. LLM predictions are improved by including the list of job titles in the prompt, but they’re still significantly worse than the CAREER model. Part of this poor performance is due to models assigning mass to occupational titles that do not exist (i.e., $\sum_{y\in\text{all jobs}}\hat{P}(\text{title}_{y}\mid\text{prompt})$ is far less than 1); however, explicitly removing this mass by renormalizing a model’s predictions does not make up a large difference. Readers can refer to Appendix G for more details on our experiments with baseline language models.

Our first strategy is to fine-tune LLMs to predict occupational trajectories on survey data. Fine-tuning on survey data would encourage models to make more representative predictions while retaining the knowledge they acquired during pre-training. Since LLMs make predictions at the token level, we fine-tune models to predict each token of a textual summary of worker careers. Specifically, we randomly divide each dataset into 70/10/20 train/validation/test splits. Splits are constructed at the individual level; if an individual is in a split, all of their observations are in the same split. We use the same test splits as for the exercises in Section 4. We then create a text template for each individual consisting of all of the observations of their career, as the second example template illustrated in Section 4.

We fine-tune the three Llama-2 models used in Section 4 on the training set text templates and evaluate models on the test split as in Section 4. Figure 2 illustrates the fine-tuning procedure. We perform fine-tuning by maximizing a model’s assigned likelihood to the true next token conditional on all previous tokens in a text template. Our objective includes each token of each template, regardless of whether or not it’s an occupation title (we do not include the full list of occupations in the prompts; as we will show later, fine-tuned models indeed learn the set of valid job titles). We perform full parameter and full precision fine-tuning for 3 epochs with a batch size 32. To improve computational efficiency for inference, we quantize fine-tuned language models to 8-bits. In Appendix E, we show that running model inference in full precision does not significantly improve performance.

Table 2 reports the test set perplexity of the three fine-tuned Llama-2 LLMs along with two baselines trained on the training split: a bi-gram Markov model that only predicts an individual’s next job from the empirical frequency of transitions, and CAREER (Vafa et al., 2024), a foundation model designed to make representative predictions on survey data. Fine-tuned models make significantly more representative predictions than the original models Table 1.

Surprisingly, the fine-tuned LLMs make more representative predictions than CAREER, which was trained on 24 million resumes and designed specifically to make accurate predictions on survey data. Although the Llama-2 models are not explicitly trained to model occupations, the information they acquire about career trajectories in the training process enables them to outperform CAREER. While CAREER is trained on proprietary resume data, the pre-trained Llama-2 models are open-source, making it possible for practitioners with computational resources to build state-of-the-art models.

It is worth noting that two models’ perplexities on the same observation are often correlated; we use a bootstrap method to better understand how significantly and consistently our models outperform CAREER. Table 3 compares the performance of different variants of fine-tuned Llama-2 models and the previous CAREER transformer by analyzing the perplexity differences in pairs of models. Specifically, we generated 1,000 bootstrap samples from each of the three survey datasets; then, we computed two perplexities on each bootstrap sample using CAREER and one of our fine-tuned Llama-2 models. Readers can refer to Appendix I for visualizations and more comparison results.

Since LLMs make predictions on the token level, they may place mass on job titles that do not exist. In Appendix H, we show that fine-tuning encourages models to assign only mass to existing occupations. For example, the fine-tuned Llama-2 models place an average of 99% of their mass on valid occupations. As a result, renormalizing the predictions of fine-tuned LLMs to ensure that they only place mass on real occupations has little effect.

While the fine-tuning approach is effective for generating representative predictions, it is computationally expensive. Here, we consider another approach with a lower computational cost. Our approach is based on passing in a text description of an individual’s career to an LLM and extracting the model’s embedding. A new classification model is trained on top of the embedding to predict the individual’s next job.

We first convert each job sequence to text using the template discussed in Section 4.1. When we pass in a text template to a language model, the model embeds the text in $d$-dimensional Euclidean space to predict the individual’s next job. To predict an individual’s next job from their embedding, we train a multi-class classifier using multinomial logistic regression. Because we are training new models on top of embeddings, we are no longer constrained to make predictions on the token level. So we train the classifier to predict occupation codes directly rather than the job title. Crucially, this approach only requires performing inference steps for each prediction (i.e., to build the emebdding), and so it is more computationally efficient than fine-tuning model parameters. See Appendix J for more details.

To extract embeddings from the Llama-2 models (fine-tuned and off-the-shelf), we use the final-layer model representation of each model. We consider both the off-the-shelf Llama-2 models considered in Section 4 and the fine-tuned models described above. We also consider an approach that takes advantage of the in-context learning capabilities of LLMs. Specifically, in addition to including an individual’s job trajectory in the template, we also include the complete trajectories of three randomly sampled individuals at the beginning of the prompt. See Appendix K for more details.

In addition to extracting embeddings from LLMs, we also consider models that are designed specifically to provide embeddings. Specifically, we generated text embeddings using three models available from OpenAI: text-embedding-3-small, text-embedding-3-large, and text-embedding-ada-002. For each survey, we report results for the model with the best performance.

Table 4 contains the results summarizing model performance. All models form far better predictions for the survey population than the original LLMs. However, there is still a substantial gap between these models and the fully fine-tuned LLMs, pointing to the importance of fine-tuning for generating representative predictions. The embeddings extracted from the LLMs form better predictions than those from the embedding-only models. In-context learning also appears to provide additional benefits at minimal computational cost. It’s worth noting that the 70B parameter models form worse predictions than the 13B parameter models. However, this may be due to regularization challenges; the 70B parameter model contains 8,192 embedding dimensions compared to 5,120 for the 13B parameter model.

Table 5 compares various approaches for predicting the next occupation in a career trajectory, evaluating them based on fixed cost, variable cost per observation, and representativeness. Classical methods such as logit models have relatively low fixed computational costs for model estimation; the variable cost per prediction is also extremely low. Since these models are not capable of capturing complicated temporal dependencies in job histories, their predictions are only moderately representative of survey datasets. Non-LLM deep learning approaches, like CAREER, require significant model training and hyperparameter tuning effort. However, they demonstrate more representative predictions compared to classical methods. The bottom three rows summarize the methods described in this paper. While using LLMs out-of-the-box requires no additional training, their predictions are poor. Fine-tuning these models results in the most representative predictions, albeit at a high fixed computational cost. The embedding approach alleviates the cost, yet the representativeness of predictions suffers.

One of the most surprising results is that fine-tuned LLMs form better predictions of occupational trajectories than CAREER, a model designed specifically to form representative predictions. CAREER was trained on a proprietary dataset of 24 million resumes. In contrast, Llama-2 is only trained on publicly available data, and it is open-source. The success of the Llama-2 model lowers the barrier to entry for researchers interested in studying career trajectories, as they no longer need access to proprietary datasets or significant computational resources to train deep learning models from scratch.

We start with inspecting models’ performance on the binary task of whether an individual will change her job (i.e., $y_{i,t}\neq y_{i,t-1}$) or not. Specifically, we define $\text{stay}_{i,t}=\mathbf{1}\{y_{i,t}=y_{i,t-1}\}$. The model predicts if an individual will stay in the same occupation with the following probability:

and define $\text{move}_{i,t}=\mathbf{1}\{y_{i,t}\neq y_{i,t-1}\}$; the predicted probability of moving is therefore

We exclude the first record $t=1$ for every individual from our analysis.

Figure 5 shows that the fine-tuned Llama-2 (70B) performs better on movers uniformly across the three datasets. However, we also note that it assigns a lower overall probability of staying in the same job. In this part, we investigate whether fine-tuned Llama-2 performs better on movers because of its tendency to allocate more probability mass to job changes in general, rather than its ability to accurately predict the specific job an individual transitions to, conditional on them moving to a new job.

To assess model performance for movers, we compute the probability of the next occupation conditional on moving:

In Table 7, we further compute the differences in model perplexity between our fine-tuned Llama-2 models and CAREER using bootstrapping. We see that the fine-tuned Llama-2 (70B) outperforms all other models. We note that the perplexity measured on the conditional modeling problem in Table 6 is much higher than the perplexities reported in Table 16 in the experiments section, which is why we also report higher differences between these models.

We explore the advantages of LLMs in predicting future occupations by asking the following question: for what kind of observations $(y_{i,t},x_{i},x_{i,\leq t},y_{i,<t})$ do language models outperform the previous specialized transformer?

We define our prediction target as the difference in the log-likelihood of the ground truth between predictions from the fine-tuned Llama-2 (70B) and CAREER. To analyze the heterogeneity in this prediction target, we employ a cross-fitting approach. First, we split the test set into ten folds. We then loop through the folds, where in each case, one of the folds is a held-out quintile evaluation fold, while the complement, the training folds, are used to construct a mapping from feature $X_{it}$ into quintiles. Using the data from the training folds, we train a regression forest (Athey et al., 2018b). We then rank the predicted values in the training fold and determine the thresholds for quintiles; this in turn determines a mapping from features into quintiles. Next, we apply this trained function to the data points in the evaluation fold, assigning each point to its corresponding quintile. We then calculate the mean value of the prediction target within each quintile using the assigned points. This process is repeated for all folds, where in each fold, the within-quintile means are estimated using an evaluation fold that is distinct from the data used to estimate the quintile mapping for that fold. Finally, we present the mean values per quintile averaged across all folds. The presence of heterogeneity in these quintile-level means, estimated on held-out data, indicates that the intensity of differences in performance between fine-tuned Llama-2 (70B) and CAREER vary as a function of the features $X_{it}$. Then, we show the values of each of several features in each quintile, allowing us to understand the factors that vary systematically between higher and lower quintiles. We conduct this analysis separately for two prediction scenarios: binary move vs. stay, and job choice conditional on moving. ²²2This method used to measure prediction heterogeneity is closely related to existing methods for analyzing heterogeneous treatment effects (HTE) (Athey et al., 2018b) where conditional average treatment effects (CATEs) are estimated from observational or randomized data where units are exposed to treatment at random. However, unlike in traditional CATE estimation, where only one potential outcome is observed for each unit, we observe both counterfactuals (i.e., the log-likelihoods from fine-tuned Llama-2 (70B) and CAREER) for every observation. In this context, the ”treatment” is the use of fine-tuned Llama-2 (70B) instead of CAREER for prediction, and the ”treatment effect” is the difference in log likelihood between the two models. The Conditional Average Treatment Effect (CATE) represents the expected treatment effect given a set of features $X_{it}$. By sorting observations into quintiles based on estimated CATE, we can assess whether there is detectable heterogeneity in the treatment effect as a function of the features. If the CATE estimates within each quintile group, which are estimated on held-out data, exhibit monotonicity, this implies that there is significant heterogeneity in the treatment effect that can be explained by the features (Chernozhukov et al., 2023). Furthermore, by examining heatmaps of feature values across the CATE quintiles, we can identify which features are associated with larger or smaller improvements in prediction performance when using fine-tuned Llama-2 (70B) compared to CAREER. This analysis allows us to interpret the sources of heterogeneity in the treatment effect and understand which features drive the differences in model performance. The fact that we observe both counterfactuals for each observation strengthens the validity of our heterogeneity analysis, as it eliminates the need for assumptions typically required in CATE estimation when only one potential outcome is observed.

We craft features $X_{it}$ from $(y_{i,t},x_{i},x_{i,\leq t},y_{i,<t})$ and use generalized random forests to discover heterogeneity in the space of $X_{it}$ with different treatment effects (i.e., the performance gap between fine-tuned Llama 2 and CAREER).

Since the feature set $X_{it}$ heavily depends on previous occupations, we exclude the first prediction ($t=1$) of each worker from our analysis. We also merge observations from test splits of the three survey datasets, analyze them together, and retain a dataset indicator.

Specifically, the feature $X_{it}$ includes:

• •

  rank: The rank of the job title $y_{i,t}$ within the individual’s career trajectory, which is the integer $t$.

• •

  job_freq: The number of occurrences of occupation $y_{i,t}$ in the dataset.

• •

  prev_job_freq: number of occurrences of occupation $y_{i,t-1}$ in the dataset

• •

  job_freq.prev_job_freq: the product of job_freq and next_job_freq

• •

  num_tokens_label: number of tokens in the next job title $y_{i,t}$.

• •

  num_tokens_prev_label: number of tokens in the previous job title $y_{i,t-1}$

• •

  num_tokens_text: number of tokens in the text representation of the job history $\mathcal{T}(x_{i},x_{i,\leq t},y_{i,<t})$

• •

  empirical_transition_freq: The empirical number of transitions $y_{i,t-1}\to y_{i,t}$, which is calculated as $\#[y_{i,t-1}\to y_{i,t}]$.

• •

  empirical_transition_prob: The empirical probability of transition $y_{i,t-1}\to y_{i,t}$, which is calculated as $\frac{\#[y_{i,t-1}\to y_{i,t}]}{\#[y_{i,t-1}]}$.

• •

  I.PrevSOC.Group...SOC.Group...approx..30.groups and
  I.PrevSOC.Group...SOC.Group...approx..30.groups: These variables are only defined for movers. We use the SOC hierarchy to cluster $y_{i,t-1}$ ($y_{i,t}$) into $\text{SOC-group}(y_{i,t-1})$ ($\text{SOC-group}(y_{i,t})$) and $\text{SOC-detailed-group}(y_{i,t-1})$ ($\text{SOC-detailed-group}(y_{i,t})$). The SOC hierarchy generates around 10 SOC groups and 30 detailed SOC groups. We add two additional indicators intuitively measuring the magnitude of job transition for movers from $y_{i,t-1}$ to $y_{i,t}$, $\mathbf{1}\{\text{SOC-group}(y_{i,t-1})=\text{SOC-group}(y_{i,t})\}$ and $\mathbf{1}\{\text{SOC-detailed-group}(y_{i,t-1})=\text{SOC-detailed-group}(y_{i,t})\}$.

• •

  dataset_indicator.NLSY79, dataset_indicator.NLSY97 and dataset_indicator.PSID - indicator variables for the dataset that each point comes from.

• •

  PCA_32: A 32-dimensional PCA representation of the full 8192-dimensional representation of the text representation $\mathcal{T}(x_{i},x_{i,\leq t},y_{i,<t})$. While these features are not easily interpretable, they help discover heterogeneity, and we only analyze the interpretable features in the subgroups in our analyses.

Previous sections have shown that our new language-based approach models our target population better than previous state-of-the-art CAREER models. In this section, we explore how models perform on different subgroups defined by educational backgrounds. Table 8 presets the perplexity differences between fine-tuned Llama-2 (70B) and CAREER on different subgroups in different datasets, suggesting that our language-based approach consistently outperforms the previous state-of-the-art model for different subpopulations.