Unlocking Model Insights: A Dataset for Automated Model Card Generation

Shruti Singh   Hitesh Lodwal   Husain Malwat   Rakesh Thakur   Mayank Singh

Ind

Abstract

Language models (LMs) are no longer restricted to ML community, and instruction-tuned LMs have led to a rise in autonomous AI agents. As the accessibility of LMs grows, it is imperative that an understanding of their capabilities, intended usage, and development cycle also improves. Model cards are a popular practice for documenting detailed information about an ML model. To automate model card generation, we introduce a dataset of 500 question-answer pairs for 25 ML models that cover crucial aspects of the model, such as its training configurations, datasets, biases, architecture details, and training resources. We employ annotators to extract the answers from the original paper. Further, we explore the capabilities of LMs in generating model cards by answering questions. Our initial experiments with ChatGPT-3.5, LLaMa, and Galactica showcase a significant gap in the understanding of research papers by these aforementioned LMs as well as generating factual textual responses. We posit that our dataset can be used to train models to automate the generation of model cards from paper text and reduce human effort in the model card curation process. The complete dataset is available on OSF.

1 Introduction

Model cards (representative sample in Table 1) were proposed by Mitchell et al. (2019) to document the training details and intended usage of models and democratize the model development process. However, significant efforts are required to create the model card as several specific details about the model need to be extracted and organized. In recent years, the documentation of models and datasets has gained significant attention due to the rapid influx of newly proposed models and datasets. Recently, some conferences such as NeurIPS (Denton et al., 2023) mandate the submission of datasheets for datasets, and a majority

Table 1: A representative model card documenting information for the BERT-BASE-CASED model.

Model Name: BERT-BASE-CASED

Train Data: BooksCorpus (8B words) & English Wikipedia (2.5B words)

Infrastructure: 4 Cloud TPUs in Pod configuration (16 TPU chips)

...

Train Objective: MLM and NSP

of conferences (EMNLP (Bouamor et al., 2023), AAAI (Chen and Neville, 2023), ACL (Boyd-Graber et al., 2022)) mandate discussion of reproducibility checklist, limitations and ethical considerations of the work. This highlights the importance of documenting datasets and methodology for future consumers of these artifacts. However, submission of these artifacts is restricted to specific venues and not mandated by all venues. Recently, huggingface added model cards for popular models manually (Mitchell, 2023). However, manual construction of model cards is a time-consuming process, and it is difficult to manually update model cards of plethora of models submitted everyday. As a result, model cards for the majority of existing models do not exist or are incomplete.

We propose a dataset in the format of question-answers that can be used to train models to generate model cards. We create a set of twenty general questions that seek relevant details about models (Table 2). We provide answers for 25 models extracted from the research papers. Our dataset can be used to train models that can extract information from research papers and automatically generate these model cards, saving time. Our dataset differs from existing QA datasets for academic papers (Saikh et al., 2022; Dasigi et al., 2021; Jin et al., 2019; Pappas et al., 2018; Tsatsaronis et al., 2015) as our question-answer pairs are specifically targeted at generating the model cards.

We evaluate large language models (LLMs) suchThe diagram illustrates a three-stage annotation pipeline. It begins with a box labeled 'QUESTIONS' containing Q1, Q2, ..., Q20. An arrow points from this box to a 'PRELIMINARY ANNOTATION' stage. This stage contains three parallel paths, each representing a model (MODEL 1, MODEL 2, MODEL 3). Each model takes a question (Q1) and produces a preliminary answer (A1). An ellipsis indicates more models. Arrows from these preliminary answers point to an 'EXPERT ANNOTATION' stage. This stage features an 'Expert Annotator' who reviews the preliminary answers and produces final expert answers (A1). The entire process is shown as a flow from left to right, with the final expert answers being the output of the pipeline.

Figure 1: Twenty questions are formulated to cover model details exhaustively. The annotation pipeline consists of three stages: (i) Question Formulation, (ii) Preliminary annotation, and (iii) Expert annotation.

as ChatGPT-3.5, LLaMa 7B, Galactica 125M, and Galactica 1.3B in generating the answers for model card questions in a zero-shot setting. Our experiments showcase that existing models perform poorly in generating answers. Additionally, their performance is worse if we consider factual details in the generated text. Inspected LLMs often generated memorized answers absent in the source papers and non-existent dataset names. Among the evaluated models, ChatGPT-3.5 performs the best; however, that can also be attributed to its huge size in comparison to the other probed models.

2 Dataset

In this section, we discuss the dataset statistics, curation pipeline, and annotation strategy. To construct our dataset, we focused on gathering ML model cards in a question-answering format. Each instance within the dataset comprises a pair consisting of a question and its corresponding answer, specifically addressing various aspects of the ML model. The questions (Table 2) encompass topics such as model training, architecture, problem statement, datasets used, etc. The domain of our dataset is limited to computational linguistics, and we select language models (LMs) which are successors of the transformer model (Vaswani et al., 2017). The dataset consists of 500 question-answer pairs for 25 models, with answers extracted from the research papers by the annotators.

The curation pipeline (Figure 1) has three phases: (i) Question Formulation, (ii) Preliminary Annotation, and (iii) Expert Annotation. The Question Formulation stage designs a standardized set of

Id	Question
Q1	What is the main problem statement being addressed in this paper?
Q2	What gaps in previous literature does this paper tries to address?
Q3	What are the main contributions of the paper?
Q4	Is the model proposed only for a specific domain, like code, images, specific text domains like finance, biomedical, etc? If yes, is it possible to extend the model to other domains?
Q5	What datasets and tasks is the model evaluated on?
Q6	Does the model show any bias/prejudice that is mentioned in paper?
Q7	List the limitations of the model discussed in the paper.
Q8	List the datasets on which the model was trained along with a brief summary and the size of each dataset.
Q9	List the tokenizer used alongwith the size of the vocabulary.
Q10	List the preprocessing techniques used on the dataset.
Q11	Describe the architecture details (whether it is encoder-decoder, encoder-only, or decoder-only framework, number of layers, number of heads, embedding dimension, total parameters). In case multiple models of varying sizes are trained, list details for all configurations.
Q12	Describe the training setup (e.g., learning rate, steps, epochs, optimizer, etc.)
Q13	Describe the computational resources used to train the model.
Q14	Are all details necessary to reproduce the paper provided?
Q15	What is the pretraining objective of the model?
Q16	What is the loss function that is used to train the model?
Q17	Consider the transformer model as the base architecture for encoder-decoder (ED) models. Similarly, consider BERT and GPT as the base for encoder-only (E) and decoder-only (D) architectures. How is the architecture of this paper different from the base architectures transformer or BERT, or GPT (depending on ED, E, or D respectively)?
Q18	What experiments are conducted in the paper? Provide a brief summary of each experiment by commenting on the task description, input, expected output, and evaluation metric.
Q19	Are ablation studies conducted in the paper? If yes, which parameters are included in the ablation study?
Q20	List the future work mentioned in the paper.

Table 2: Question set used for extracting model card details.

twenty questions that cover important aspects of the model, such as training details, architecture, problem statement, and model bias. These questions offer valuable insights into the model. We utilized the same question set for all the models included in our final collection of model card QA pairs. The complete set of questions is provided in Table 2. In Preliminary Annotation, we curate a list of 30 popular LMs such as Longformer (Beltagy et al., 2020), Transformer-XL (Dai et al., 2019), BART (Lewis et al., 2020), etc.; and employ 25 annotators to select a model of their preference from the model list. The 25 annotators extract the preliminary answers for different models from the re-spective research papers. The annotators are undergraduate and graduate students who have in-depth knowledge of traditional ML and basic knowledge of DL, including transformer architectures such as BERT. In Expert Evaluation, a subject expert with expertise in the field of DL (masters student in CS with prior experience in DL architectures and frameworks) reviews the answers extracted in the preliminary annotation stage. They examine the papers and assess the answers for accuracy, completeness, and relevance. Their expertise allows them to identify any inaccuracies or inconsistencies in the preliminary annotation stage answers and provide accurate assessments. By incorporating both the preliminary and expert annotation stages, the curation pipeline adds an extra layer of annotation and combines different perspectives and expertise levels to establish a comprehensive and reliable ground truth dataset for model cards. All the annotators are provided with an annotated example for BERT (included here). The annotators were instructed to extract complete answers from the research paper. The answer can span multiple sentences and paragraphs.

3 Benchmarking LLMs for Model Card Generation

Recent developments in instruction-following LMs (Ouyang et al., 2022; Chung et al., 2022; Köpf et al., 2023; Taori et al., 2023; Chiang et al., 2023) show improved performance for downstream tasks and has led to their increased usage in QA tasks. We evaluate the performance of LLMs in zero-shot QA for generating the model cards. ChatGPT-3.5 (Wu et al., 2023) by OpenAI uses RLHF, and its training data details are publicly unavailable. LLaMa (Touvron et al., 2023) by Meta AI is trained on trillions of tokens from public data, including arXiv L^AT_EX files, Github, and Wikipedia. Galactica (Taylor et al., 2022) is a scientific LM by Meta AI, trained on a 106 billion tokens data including research papers, scientific KBs, and Github (detailed discussion in appendix).

3.1 Prompting LLMs for QA

We evaluate ChatGPT-3.5, LLaMa (7B parameters), Galactica (125M parameters), and Galactica (1.3B parameters) in a zero-shot setting to test their ability to generate answers for model card questions. The subject expert annotator prompts ChatGPT for a particular model in a single ses-

The diagram illustrates the prompting process for ChatGPT-3.5. It consists of three sequential steps, each with a prompt box and a response box. The first step shows a prompt asking for paper details (Title, Authors, Venue, Year, Arxiv URL, Other URLs) and a response box with a placeholder. The second step shows a prompt asking for the main problem statement and a response box with a placeholder. The third step shows a follow-up question and a response box with a placeholder. Icons of a person and a robot are used to denote the user and the model, respectively.

Figure 2: Prompting ChatGPT-3.5 for generating answers for model card questions.

sion, starting with a general prompt trying to elicit model details. Figure 2, showcases the prompting procedure employed to generate answers. Based on the ground truth, the subject expert also marks the ChatGPT-3.5 answer into correct and incorrect spans. For LLaMa evaluation, we download the original LLaMa 7B weights made available by Meta AI and then use the llama.cpp repository (Gerganov, 2023) to convert it to a 4-bit integer quantized model. We directly use the Galactica 125M and Galactica 1.3B models made available by Meta AI to generate the answers. For LLaMa and Galactica models, we restrict the output size to 1000 tokens.

3.2 Qualitative and Quantitative Evaluation

We perform a quantitative evaluation of the LLM-generated responses by computing the BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), and BERT-Score (Zhang\* et al., 2020). We represent the BERT-Score Precision, Recall, and F1 Score as BS-P, BS-R, and BS-F1, respectively. However, these automatic evaluation metrics do not take into account the factuality of the answers. For e.g., an LLM-generated answer which exactly matches the ground truth answer but differs in training dataset name receives a high score, disregarding the inaccurate dataset name, a crucial aspect in an answer describing the training dataset. For e.g., the mBART (Liu et al., 2020) model is evaluated on the WMT-16, WMT-17, WMT-18, and WMT-19 datasets, but LLaMa generated answer says ‘mBART achieves state-of-art accuracy on WMT-14 dataset’, which is incorrect and not mentioned in the paper.

To better understand the factuality of generated answers, we do a small-scale qualitative evaluation. An expert annotator (different from the annotator

LLM-generated	Ground Truth	Label
The architecture is based on the BERT architecture, an encoder-only model.	DistilBERT has the same general architecture as BERT.	CC
The pretraining objective is same as BERT, i.e., predict some masked tokens next sentence prediction objective.	We follow the BERT training, i.e. MLM but without the	PC
The model achieves state-of-art accuracy on WMT-14 dataset.	Results indicate a new SOTA on the WMT-16 English-Romanian.	IC

Table 3: Representative examples of LLM-generate text and ground truth, marked as CC, PC, and IC. Artifact names and concepts (underlined) are crucial in determining the label.

in data curation) evaluates the LLM-generated answers for five models in the dataset for each of the LLMs (ChatGPT-3.5, LLaMa, Galactica 125M, and Galactica 1.3B). We choose three different labels, Completely Correct (CC), Partially Correct (PC), and Incorrect (IC); to evaluate the answers. If the LLM answer exactly matches the ground truth answer in terms of relevance, factuality, and exhaustively covers all crucial facts, the response generated by the LLM is marked CC. If the generated response consists of some correct and some incorrect facts, it is marked PC. If none of the facts in the generated answer are correct, the answer is marked IC. Special attention is paid to dataset, model, and task names during evaluation, and any mismatch in the names is marked as incorrect. In the earlier provided example, where LLaMa listed WMT-14 dataset while the ground truth was WMT-16, WMT-17, WMT-18, and WMT-19 datasets, the subject expert annotator marked the answer as incorrect, even though the phrase ‘WMT’ will be matched by other automatic evaluation metrics. BS-F1 for these pair of sentences is high, 0.795, even though the dataset is incorrect. We present representative examples of CC, PC, and IC in Table 3.

3.3 Results of zero-shot QA using LLMs

We showcase the average BLEU, ROUGE-L, and BERT Score P, R, and F1 in Table 5. We observe that ChatGPT-3.5 performs the best across all metrics, and LLaMa 7B performs comparably to ChatGPT-3.5. It should be noted that the models have different parameters, and hence the results should not be generalized to bigger LLaMa and Galactica models. Question-wise scores (Table 4) show that ChatGPT-3.5 achieves the highest ROUGE-L scores in answering Q6 and Q13,

questions about model biases and computational resources. Galactica 125M performed worst than all models, often generating repetitive text, which could also be attributed to its size. A major limitation of all LLMs is that their text generation is not grounded on facts. To analyze the LLM-generated answers for factuality, we present the results for qualitative evaluation in Table 6. The low scores indicate that most of the generated answers are incorrect and often memorized based on the frequency in the training dataset. For example, a manual analysis reveals that ChatGPT-3.5 often reports the training infrastructure as 3 V100 GPUs, irrespective of the model for which we probe for the training details. While ChatGPT-3.5 generated mostly partially correct responses, other models’ responses were often incorrect, with Galactica 125M having almost 92% incorrect responses.

4 Potential Usage

We posit that our dataset of QA pairs can be used to train models for generating model cards. Our dataset can be used for instruction tuning LLMs for generating ML model cards by prompting specific questions about different aspects of the model. Similarly, our dataset of answers generated from LLaMa and Galactica models can be utilized as there are initial answers from LLMs and ground truths. So LLMs can be prompted with the whole conversation, where they are provided with a question, incorrect answer, and then pointed the correct answer. For ChatGPT-3.5-generated responses, we provide another layer where we label which spans of the answer are incorrect.

5 Related Work

Information Seeking Datasets for Research Papers:

Previous works curate datasets from research papers for fact verification and facilitating paper reading. ScienceQA (Saikh et al., 2022) comprises 100k synthetic question-answer-context triples, where the questions are generated from a filtered set of noun phrases extracted from 1825 IJCAI papers. QASPER (Dasigi et al., 2021) is a dataset of 5049 questions over 1585 NLP papers, with extractive, abstractive, and binary yes/no answers. Multiple works curate QA datasets in the biomedical domain, such as BIOASQ Tsatsaronis et al. (2015), BioRead Pappas et al. (2018), and PubMedQA dataset (Jin et al., 2019). SCIFACT (Wadden et al., 2020) dataset consists of 1.4k

	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Q11	Q12	Q13	Q14	Q15	Q16	Q17	Q18	Q19	Q20
ChatGPT-3.5	0.15	0.14	0.18	0.23	0.15	0.3	0.18	0.24	0.19	0.19	0.23	0.22	0.27	0.22	0.21	0.2	0.23	0.19	0.17	0.21
LLaMa 7B	0.15	0.23	0.18	0.2	0.1	0.15	0.14	0.21	0.17	0.2	0.19	0.17	0.06	0.15	0.1	0.26	0.17	0.13	0.19
Galactica 125M	0.11	0.14	0.1	0.11	0.09	0.07	0.11	0.25	0.1	0.12	0.17	0.1	0.09	0.04	0.11	0.05	0.18	0.08	0.05	0.11
Galactica 1.3B	0.2	0.18	0.16	0.16	0.13	0.13	0.14	0.27	0.15	0.14	0.22	0.16	0.18	0.09	0.1	0.09	0.22	0.13	0.13	0.14

Table 4: ROUGE-L scores of the LLMs in generating the answers for the model card for each question.

	BLEU	RO-L	BS-P	BS-R	BS-F1
ChatGPT-3.5	0.0230	0.2052	0.5785	0.5638	0.5679
LLaMa 7B	0.0083	0.1958	0.4972	0.5298	0.5086
Galactica 125M	0.0079	0.1417	0.4213	0.5004	0.4493
Galactica 1.3B	0.0072	0.1559	0.3981	0.5112	0.4374

Table 5: Scores of the LLM-generated answers for the model card questions using BLEU, ROUGE-L (RO-L), BS-P, BS-R, and BS-F1. Best scores are underlined, and second-best are in bold.

	CC	PC	IC
ChatGPT-3.5	25.0	43.0	32.0
LLaMa	0	17.0	83.0
Galactica 125M	0	8.0	92.0
Galactica 1.3B	0	15.0	85.0

Table 6: Percentage CC, PC, and IC answer evaluated on five ML models (20 questions each, total 100 QA pairs). The majority of LLM-generated answers are IC or PC.

expert-written biomedical scientific claims and evidence abstracts with labels (support or refutes) and rationales. In comparison to the existing datasets, our dataset is highly specific and curates model card information for an efficient understanding of ML models.

Model cards and allied concepts: Similar to Model cards (Mitchell et al., 2019), datasheets for datasets (Gebru et al., 2021) are proposed to document dataset information and bridge the gap between dataset creators and consumers. Datasheets are proposed to document the data collection process, sources, intended use cases, etc., to promote transparency. RISKCARDS (Derczynski et al., 2023) document the risks associated with LMs by constructing a detailed skeleton to document risks such as harm type (which group is at what type of harm), references, and sample prompts and LM output. AI Usage Cards (Wahle et al., 2023) is proposed to standardize the reporting of the usage of AI technologies such as ChatGPT-3.5. These works emphasize the need for documenting artifacts in this age of information overload. We posit that our dataset will be utilized by models to assist in automating model card generation. This has positive prospects as instruction tuning has shown

promise in improving the ability of LLMs to follow instructions and show significant improvement in downstream tasks. Apart from our QA dataset, we also provide the QA pairs generated by various LLMs such as ChatGPT-3.5, LLaMa, and Galactica. Lastly, model cards for some models are available on huggingface; however, those are unstructured and available in free-form text. Our dataset provides structured model card details in a QA format that can be leveraged by LLMs to learn model card generation. There are no ethical concerns with our dataset and it doesn’t contain offensive content or personally identifiable information.

6 Conclusion

To summarize, we curate a QA dataset of 500 pairs from research papers of 25 machine learning models. A set of 20 general questions are curated for the purpose of seeking crucial details of model training, dataset, problem statement, ablation studies, etc. We employed a two-stage annotation pipeline consisting of preliminary and expert annotation stages to ensure the quality of the curated dataset. Next, we evaluate the capability of LLMs ChatGPT-3.5, LLaMa, and Galactica in generating answers for the model card questions. All the evaluated models include infactual details in the answer, highlighting the potential for developing better models for model card generation. In the future, we plan on expanding the question set and covering models from other domains, such as CV and robotics.

References

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Houda Bouamor, Juan Pino, and Kalika Bali. 2023. The 2023 conference on empirical methods in natural language processing. Accessed: June 15, 2023.

Jordan Lee Boyd-Graber, Naoaki Okazaki, and Anna Rogers. 2022. The 61st annual meeting of the association for computational linguistics. Accessed: June 15, 2023.Yiling Chen and Jennifer Neville. 2023. The 37th aaai conference on artificial intelligence. Accessed: June 15, 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%\* chatgpt quality.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Association for Computational Linguistics.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599–4610, Online. Association for Computational Linguistics.

Emily Denton, Jung-Woo Ha, and Joaquin Vanschoren. 2023. Neurips datasets and benchmarks 2023. Accessed: June 15, 2023.

Leon Derczynski, Hannah Rose Kirk, Vidhisha Balachandran, Sachin Kumar, Yulia Tsvetkov, MR Leiser, and Saif Mohammad. 2023. Assessing language model deployment with risk cards. arXiv preprint arXiv:2303.18190.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92.

Georgi Gerganov. 2023. llama.cpp.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023. Openassistant conversations—democratizing large language model alignment. arXiv preprint arXiv:2304.07327.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Margaret Mitchell. 2023. Model cards on huggingface. Accessed: June 15, 2023.

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY, USA. Association for Computing Machinery.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Dimitris Pappas, Ion Androutsopoulos, and Haris Papa-georgiou. 2018. BioRead: A new dataset for biomedical reading comprehension. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. Scienceqa: a novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang,and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford\_alpaca.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16(1):1–28.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics.

Jan Philip Wahle, Terry Ruas, Saif M. Mohammad, Norman Meuschke, and Bela Gipp. 2023. Ai usage cards: Responsibly reporting ai-generated content. In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL).

Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136.

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

A Appendix

A.1 Dataset Statistics

Our dataset consists of 500 examples, i.e. twenty question-answer pairs for 25 language models. The list of language models is presented in Table 7.

Language Models
MBART	DistilBERT	Longformer
Sparse Transformer	ERNIE	PEGASUS
Reformer	BART	RoBERTa
BigBird	MobileBERT	GPT2
ELECTRA	StructBERT	MuRIL
GPT	SpanBERT	ALBERT
XLNet	BERT-PLI	Transformer
T5	FNet	TransformerXL
SciBERT

Table 7: List of language models covered in the model card dataset.

Table 8: The dataset consists of 20 QA pairs for 25 models, amounting to overall 500 QA pairs for model card generation.

Models	25
Questions per model	20
Total QA pairs	500

A.2 Benchmarking LLMs for Model Card Generation

In Figure 3, we present the box plots of scores of the LLaMa 7B, ChatGPT, Galactica 125M, and the Galactica 1.3B model. We compare the BLEU, Rouge-L, and BERT-Score of the evaluated models. The plot indicates that the median ROUGE-L scores for LLaMa 7B and the ChatGPT model are better than the Galactica models. ChatGPT outperforms the other models in terms of BERT-Score. The BLEU scores for all the models denote poor performance, with scores almost close to zero, indicating significant scope for improving these models for the model card generation task.

Figure 3: Distribution of BLEU, ROUGE-L, and BS F1 for ChatGPT-3.5, LLaMa 7B, Galactica 125M, and Galactica 1.3B.### A.3 Large Language Models

ChatGPT-3.5 (Wu et al., 2023), developed by OpenAI, is a sibling to the InstructGPT (Ouyang et al., 2022), which uses human feedback to align outputs of language models with the human intent. While the training dataset detail of ChatGPT-3.5 is not publicly available, the model is trained with reinforcement learning with human feedback on a dialogue dataset.

LLaMa (Large Language Model Meta AI) (Touvron et al., 2023) is an LLM developed by Meta AI in October 2021, available in sizes ranging from 7B to 65B parameters. LLaMa is trained on trillions of tokens, using publicly available datasets such as C4, CommonCrawl, Github, Wikipedia, Gutenberg and Books3, arXiv L^AT_EX files, and StackExchange.

Galactica (Taylor et al., 2022) is a scientific language model by Meta AI, trained on a 106 billion tokens dataset. Galactica is trained on a diverse dataset of research papers, reference materials from encyclopedias, textbooks, Wikipedia, StackExchange, scientific knowledge bases, scientific and academic Common Crawl, and academic Github code repositories.